Characterizing states of subject

ABSTRACT

Among other things, a user of a browser is exposed simultaneously to three interfaces: A viewing interface for at least one image of a subject that is stored on a device on which the browser is running, a decision support interface that aids the user in determining the state of the subject based on the image, and a template interface that aids the user in capturing uniform descriptive information about the state of the subject. At least two of the viewing interface, the decision support interface, and the template interface operate cooperatively so that actions of the user with respect to one of the two interfaces causes changes in content exposed by the other of the two interfaces.

RELATED APPLICATIONS

This application is a continuation and claims priority to U.S.application Ser. No. 14/272,021, filed May 7, 2014, the entire contentsof which application is hereby incorporated by reference.

BACKGROUND

This description relates to characterizing states of subjects.

In diagnosing a medical condition of a patient, for example, a doctorcan apply her experience and knowledge to health-related informationabout the patient (for example, age, weight, body temperature, symptoms,medical history) to classify the medical condition. Frequently, thehealth-related information includes images captured by a variety ofmodalities (x-ray, computed tomography, magnetic resonance imaging).Correlations exist between health related information about populationsof patients and medical conditions of those patients that enable anexperienced doctor to make an accurate diagnosis. It is commonplace forthe doctor then to create a medical report that captures thehealth-related information and the diagnosis.

Traditionally medical reports are created by hand on paper and stored inpaper files associated with the patients. The language used on medicalreports to describe medical conditions has become standardized in manycases, and skilled doctors are accustomed to making careful use of thestandardized language so that other people who read the reports willfind them clear and understandable.

Computer-based systems have been developed to aid medical professionalsin accumulating, storing, and analyzing health-related information aboutpatients and in producing clear and accurate medical reports quickly andeasily. Among other things, there are templates available that help thedoctor to consistently use well accepted terminology and to producemedical reports that are complete and accessible quickly to others.

The quality and accuracy of the classification and characterization ofstates of subjects varies widely depending on the skill, experience,education, training, and other factors associated with the people doingthe classification and characterization. For example, in some areas ofthe world, pathologists perform their work less effectively, lessefficiently, and less accurately than in other areas of the world, duelargely to less adequate or suboptimal educational opportunities, amongother things.

SUMMARY

In general, in an aspect, a user of a browser is exposed simultaneouslyto three interfaces: A viewing interface for at least one image of asubject that is stored on a device on which the browser is running, adecision support interface that aids a determination of the user about astate of the subject based on the image, and a template interface thataids the user to capture uniform descriptive information about the stateof the subject. At least two of the viewing interface, the decisionsupport interface, and the template interface operate cooperatively sothat actions of the user with respect to one of the two interfacescauses changes in content exposed by the other of the two interfaces.

Implementations may include one or a combination of any two or more ofthe following features.

The template interface includes interactive controls that representtemplated structured content; interaction with the interactive controlsis reflected in a digitally stored record associated with the subject.The subject includes a person, and the state includes a medical state ofthe person. The browser includes Google Chrome. The interfaces aredevice independent and operating system independent. The subjectincludes a person, and the uniform descriptive information includes partof a medical report. The subject includes a person, and the imagesinclude images generated by at least one medical imaging modality. Theimages are downloaded by the browser and stored on local storage. Theimages are DICOM compliant. The images stored on the device have beenreceived from a server. The template interface has at least oneinteractive user feature, and the tool of the viewing interface issensitive to the feature.

The viewing interface includes at least one tool that aids the user ininteracting with the images. The tool is context-sensitive. The contextto which the tool is sensitive includes an imaging modality or a part ofthe subject that is imaged or both. The template interface providesinteractive controls that represent selections, or choices, ormodifiers, and interaction with the controls governs at least a part ofthe descriptive information that is captured about the state of thesubject. The viewing interface includes at least one tool that aids theuser in understanding the images, and the tool is contextually sensitiveto user interaction with the interactive controls of the templateinterface. The viewing interface includes tools that aid the user inunderstanding the images, and the tools are contextually sensitive touser interaction with the interactive user controls of thedecision-support interface. The tool has a feature that is sensitive toat least one context of at least one of the interfaces exposed to theuser. At least one image is associated with a part of the subject, andthe feature of the tool is sensitive to the part of the subject. Atleast one image was captured using a particular modality, and thefeature of the tool is sensitive to the modality.

The decision-support interface displays at least one of a diagram, animage, or a text element associated with states of subjects, andincludes interactive user controls. The decision support interface hasat least one interactive user feature, and the tool of the viewinginterface is sensitive to the feature.

The tool represents at least one normal value that is sensitive to anattribute of an image to which the tool may be applied. The tool has afeature that aids the user in using the tool. The feature includes avisible indicator on the viewing interface. The feature is provided inthe decision support interface. The feature in the decision supportinterface includes at least one normal or abnormal example or diagram. Ause of the tool causes at least one value associated with the tool to beassociated with a feature of the template interface. The use of the toolincludes a dragging and dropping of the tool or an image to which thetool has been applied in the viewing interface, into the templateinterface of the decision support interface. The use of the toolincludes a context sensitive application of the tool to the image.

At least one of the images is associated with stored attributes, and useof a feature of the template interface causes an annotation of thestored attributes of the image. At least one of the images is associatedwith stored attributes, and use of a feature of the decision supportinterface causes an annotation of the stored attributes of the image. Atleast two of the interfaces are associated with stored attributesassociated with images, templates, or decision support elements, and atleast some of the attributes are expressed as objects that includevalues and methods. Values of at least some of the attributes arechanged in response to actions by the user in using at least one of theinterfaces.

In general, in an aspect, information is stored about images of asubject, content of a report is stored that includes uniform descriptiveinformation about a state of the subject, and elements of a decisionsupport system are stored that are useful in determining the state ofthe subject. The information is stored as programming objects thatcontain values of attributes and methods associated with the attributes.The value of at least one of the attributes of one of the objects can bechanged by executing a method of one of the objects based on interactionof a user with one of the images, with a template for the report, orwith an element of the decision support system.

In general, in an aspect, a user is exposed, through an interface of abrowser, to a facility for use in determining a state of a subject,based on (a) information presented to the user about attributes of thesubject and (b) information about relationships between states of othersubjects and attributes of the other subjects. The facilityinteractively guides the user in determining the state of the subject bypresenting interactive controls representing possible attributes of thestate of the subject to the user. The interactive controls and thefacility respond (1) to interaction by the user with the controls and(2) automatically to information about relationships between states ofother subjects and attributes of other subjects provided from the app orthrough the browser. The interaction by the user with the controlsincludes touching a display. The facility exposed to the user throughthe interface enables the user to generate a report determining thestate.

Implementations may include one or a combination of any two or more ofthe following features. Determining the state of the subject includesclassifying the state of the subject by classifying informationassociated with the subject based on normative aggregated informationassociated with other subjects. Classifying the state of the subjectincludes diagnosing a medical state of a person. The informationpresented to the user includes images associated with the subject. Theinteractive controls include text phrases that can be part of a textreport of the state of the subject. The interaction by the user with thecontrols includes the user indicating features of the state of thesubject based on the information presented to the user about theattributes of the subject. The relationships between the states of othersubjects and attributes of the other subjects are based on expertknowledge. The relationships between the states of other subjects andattributes of the other subjects are part of a decision support system.

Information is presented to the user through the interface about therelationships between the states of other subjects and attributes ofother subjects using diagrams and examples, e.g., of prior subjects. Theinteractive controls representing possible attributes are presented in astepwise manner representing expert knowledge. Expert knowledge isembedded in the information about relationships between states of othersubjects and attributes of the other subjects, and the expert knowledgeis made available to the user based on a context implied by Interactionsof the user with the facility used in determining the state of thesubject.

The interactive controls representing possible attributes are presentedin at least two different modes suitable respectively for users ofdifferent skill levels. The interactive controls representing possibleattributes are presented in a collapsed mode for a user who candetermine the state of the subject without requiring presentation ofdecision support information. The collapsed mode enables the user togenerate a report of the state of the subject by noting only what isabnormal or requires a comment. The interaction by the user with thecontrols includes dragging and dropping images that represent attributesof the subject onto text associated with the controls. The images areautomatically annotated with attributes based on the text onto which theimages are dropped. Dropping of an image onto text associated with thecontrols causes the text associated with at least one of the controls toexpand to include additional uniform text representative of knowledgeprovided by a subject matter expert. The interactive controls areassociated with attributes of the subject and the interactive controlsinclude links to decision support elements associated with theattributes.

The decision support elements include diagrams or examples. A user caninvoke one or more of the decision support elements and, in response toan invocation of one of the decision support elements, correspondinginteractive controls of the state determining facility are updatedautomatically. Decision support elements are automatically appended to areport generated by the state determining facility.

A second user can interact with the interactive controls of the statedetermining facility to change interactions of the first user, anddifferences between the two interactions can be measured as anevaluation of the skill of one of the users. The attributes ofinteractive controls of the facility as set by the user and as set byother users are exposed to the user as feedback. The other users includepeople who have determined with certainty from the subject the accuracyof the determination of the state of the subject. The user can identifythrough the interface, information about the subject as representing anexample of a determination of states of similar subjects represented byelements of a decision support system, and automatically updating thedecision support system to include the example.

In general, in an aspect, a user is exposed through an interface of abrowser to a decision support facility that uses decision supportinformation to aid the user in expressing a state of a subject based onattributes of the subject. The user interface provides viewing featuresthat enable a user to identify attributes of the subject for use by thedecision support facility. The user interface provides a determinationfacility enabling the user to express the state of the subject throughinteractive controls. The decision support facility uses the user'sexpressions of the state of the subject to provide decision supportinformation.

Implementations may include one or a combination of any two or more ofthe following features. The decision support facility follows the user'sexpressions of the state of the subject step by step and presentsrelevant decision support information for each step in turn. Thedecision support facility follows the user's invocation of respectiveinteractive controls and presents relevant decision support informationfor each of the controls. The decision support information that isprovided causes automatic updating of the interactive controls. Thedetermination facility reflects information provided by a subject matterexpert based on user's expressions of the states of subjects. Thedecision support information that is provided includes examples ofnormal, abnormal, and normal variants. The decision support informationthat is provided includes indications of tools to use to captureattributes of the subject. A database of examples used by the decisionsupport facility is automatically updated to include examples identifiedby the user in using the state determination facility.

A user may drag and drop an image from the viewing interface onto thetemplate interface, from the viewing interface onto the decision supportinterface, or from the decision support interface onto the templateinterface, or any combination or two or more of those. When an image isdragged and dropped, a review is triggered to be done by a person whohas experience in understanding a relationship between the image and thestate of the subject. If information entered into the template interfaceby the user is indicative of a critical finding that requires timesensitive action, an automatic notification is sent to a party foraction. The template interface aiding the user includes displaying, inresponse to the user interaction with elements of the templateinterface, information that guides the user in selecting elements of thetemplate interface for inclusion in a report about the state of thesubject. The information that guides the user is determined by a subjectmatter expert.

Data is provided to a radiology information system (RIS) indicative oftimes associated with an image. The times include at least one of a timewhen the image is sent, a time when an analysis of the image is done,and a turnaround time. An excellence score is derived for thedetermination of the user about the state of the subject based on theimage. The excellence score is based on elapsed time, efficiency, andquality. The excellence score is based on a severity or conspicuitylevel of the state of the subject.

A report generated by the template interface is communicated to a partyother than the user who has knowledge of an actual state of the subject.The party can report information related to the actual state of thesubject, and the information reported by the party is stored for use incorrelating the determination of the state of the subject made by theuser and the information related to the actual state of the subject. Thetimes and sequence of steps of user interaction with the interfaces arerecorded for later playback.

An analysis is done of the determination about the state of the subject,structured data associated with the image, and information thatcorrelates states of subjects with the structured data, as a check ofthe likely appropriateness of the determination about the state of thesubject. The structured data includes an age of the subject, and theinformation correlates a pathology with an age range. The structureddata includes a DICOM header. The check is made at the time a user ismaking a proposed determination. Data is stored in a radiologyinformation system (RIS) that associates efficiency related timestampswith images for which corresponding users of the template interface havecaptured uniform descriptive information about the state of thesubjects. Scores are assigned to users of the template interface basedon efficiency related timestamps of images.

Elements presented by the template interface are altered to indicate toa particular user an aspect of the uniform descriptive information thatrequires attention by that user. The altering of the elements is basedon past performance of the user with respect to similar images. Thealtering of the elements includes requiring the particular user tochoose positively or negatively whether an element of the uniformdescriptive information is present or absent. The altering of theelements includes changing the color of the elements. The user isprovided real-time feedback in the template interface based oninformation about the user, the subject, or the uniform descriptiveinformation. The real-time feedback includes notifications about errorsmade by the user. The real-time feedback includes other states of thesubject commonly associated with the current state of the subject beingdescribed. For example, the real-time feedback includes notificationsassociated with syndromes or common co-morbidities suggested by theuser's capturing of certain uniform descriptive information about thestate of the subject. Uniform descriptive information associated with aright side or a left side of the subject is made available to the user.The decision-support interface displays predetermined pathologicpresentations and descriptions of the pathologic digital presentationsin words commonly used by a subject matter expert (SME). The commonlyused words depend on a stage of a developing state of the subject. Theuser can invoke the commonly used words of the subject matter expertdisplayed in the decision-support interface and causing the commonlyused words to appear in the template interface.

These, and other features, aspects, and implementations, andcombinations of them can be expressed as methods, methods for doingbusiness, software products, systems, components, apparatus, methods,and in other ways.

Other features, objects, and advantages will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION

FIG. 1 is a block diagram of a system for use in classifying structuredcontents.

FIG. 2 is a block diagram of a screen simultaneously displaying threeinterfaces.

FIG. 2A is a screen shot of two screens simultaneously displaying threeinterfaces.

FIG. 3 shows an example of a worklist.

FIG. 3A shows an example of a pop up in a worklist.

FIG. 4 shows an example of a viewer.

FIG. 5 shows an example of an active template.

FIG. 5A shows an example of a simultaneous display of a viewer and anactive template.

FIG. 5B shows an example of an active template in a guided mode.

FIG. 5C shows an example of another active template.

FIG. 6 shows an example of a template selection interface.

FIG. 6A shows an example of a drop down menu in a template selectioninterface.

FIG. 6B shows an example of a template editor.

FIG. 6C shows an example of another template selection interface.

FIG. 6D shows an example of another template editor.

FIG. 7 shows an example of a decision support system.

FIG. 8A shows an example of a preliminary report.

FIG. 8B shows an example of an active template while the report is in averified status.

Among other things, we describe here systems and techniques thatprovide, in an integrated and seamless way, features and facilities to:(1) enable subject matter experts in a field to capture, store, correct,and update their knowledge and experience about correlations betweeninformation (e.g., structured content) that is indicative of the stateof a population of subjects and the actual states of those subjects; (2)enable users to characterize (e.g., classify) the actual or believedstates of other subjects based on information (e.g. structured content)that is indicative of the states of the other subjects; (3) provide userinterfaces that govern interactivity between users and the userinterfaces in a way that implicitly represents the stored knowledge andexperience of the subject matter experts, and guides and enables theusers to achieve clear and accurate characterizations andclassifications (that conform to standard ways of expressing thecharacterizations and classifications) of the states of the subjects;(4) use characterizations produced by users with respect to states ofsubjects to improve stored information about the correlations, thecharacterizations, and classifications; (5) provide interaction andfeedback between users of the featured facilities, including users ofdifferent levels of experience and skill; (6) evaluate users based onthe quality, speed, effectiveness, clarity, and completeness of theircharacterizations and classifications; (7) educate and train the users;(8) provide a hardware and operating system independent, efficient,rapid, and easy to use way to view high-quality images anywhere in theworld and seamlessly link the user's activities with respect to theimages with the user's activities in characterizing and classifying theconditions of the subjects, and in studying information provided bydecision-support systems; and (9) enable users to translate theinformation and characterizations and classifications of the informationproduced by the users and the systems and techniques into a differentlanguage, so that the systems and techniques can be readily available inanother language.

In the course of our description, we sometimes use each of the followingterms broadly to include, for example, the categories and instancesreferred to below:

Viewer or viewer interface—a feature that presents visual data, such asimages, for viewing by a user, for example, a user who is viewing andworking with the images in connection with classifying the condition ofthe subject. In some examples, the viewer or viewer interface can beengineered to present DICOM compliant image data along with contextsensitive user tools that allow for manipulation of the image forvisualization of image findings, or to make measurements of size orother characteristics related to the images (angles, attenuation,volume, etc.) or for other purposes. In some cases, the viewer can be anactual subject, instead of visual data for the actual subject. Forexample, when an internist performs a structured physical exam on anactual patient, the patient becomes the viewer. The internist can workwith the viewer, along with other systems, e.g., a template or activetemplate for History and Physical, and a decision support system, bothof which are described below.

Decision support system—a system that supports a user in makingdecisions that characterize her classification of the conditions ofsubjects. In some examples, the system can include a knowledge base thatcontains both static knowledge related to the subject (such as anatomy,normal and abnormal findings, normal ranges related to measurements, ageof subject, position of finding within an anatomic structure) as well asdynamic data based on images (anonymized) which represent examples ofnormal and abnormal findings. In some cases, the decision-support systemcomprises a large number of stored elements or objects that representthe static knowledge and dynamic data. The elements or objects can betagged with essentially unlimited attributes which relate to theircontext sensitive presentation to the user as well as their behaviorwhen dragged into a report or clicked to produce changes in eitheractive templates or the viewer interface.

Template or active template—a set of one or more canonical sentenceseach of which contains one or more possible words or phrases in asequence such that any two or more of the words or phrases within theordered sequence can potentially form a sentence that is descriptive ofa condition of a subject. Taken together, the set of canonical sentencesrepresent statements that characterize or classify the vast majority (orall) of the findings about the condition of the subject. For example,the set of canonical sentences together might represent essentially allthe possible findings about the condition of the subject that isrepresented by a specific radiographic study, such as a study done usinga particular modality evaluating a particular body part (e.g., X-rayChest, or CT abdomen). In some cases, the template or active templateessentially represents a routine checklist that would be used by atrained person, such as a radiologist. Use of subject matter experts tobuild templates allows the templates to incorporate best practices inuse of terminology, order of report findings, and appropriatedescriptors. It also allows the effortless use of even complex toolswhich are prompted by making a simple finding. (For example, findingsmay be easy to see on an image, but remembering which angle tool to useto quantify the severity may be difficult for a generalist). The neteffect is to make even a generalist capable of rendering an expert levelreport. A template can be active, for example, if a user interactionwith the template (or other information from a decision-support systemor from a viewer) causes the template to change its form or content orboth.

Interface—a presentation (for example, visual) of information, forexample, on a computer screen, which the user interacts with to performfunctions related to their role in the system. For example, an interfacecould have three features or be comprised of three interfaces, such as aviewer, and active template, and a decision-support system. Among otherthings, in some examples, interfaces allow the user to view uniqueinformation and classify the information into structured reports throughthe use of integrated active templates and decision-support systems. Twoor three interfaces can be presented simultaneously on one or morecomputer screens. For example, a left screen of two screens arrangedside by side can show the decision support system and an active templateand a right screen can show a viewer.

Image—any kind of visual or graphical information that the user uses inclassifying or characterizing a condition of the subject. In radiology,for example, the image can be a diagnostic image, such as photographicfilms, ultrasound, CAT scan, MRI, nuclear medicine scan, and mammograms.In some implementations, these images are what are presented in thebrowser based viewer. In some implementations, the image is the visualinformation of an actual subject, e.g., an actual patient.

Tool—includes any device used within the viewer to manipulate the imagesor measure or test characteristics of the images. In someimplementations, the manipulation tools include but are not limited to:magnification, pan, window level, rotate, flip, scroll (through a stackof images—such as in CT or MRI). Also included in some cases are toolsfor measurements such as linear distance/size, volume, Hounsfield units(measurement of x-ray attenuation—roughly electron density of matter),angle(s), and any number of context sensitive measurement tools autochosen through the use of the decision-support system and activetemplates. Examples include the spine disc bulge measurement tool forspine MRI studies, and Boehler's angle of the calcaneus in a calcaneusfracture. In some implementations, for a structured physical exam, thetools can also include a blood pressure cuff. In some implementations,the tools have wireless connectivity, and the tools can automaticallypower up when the tools need to be used. For example, the blood pressurecuff can be automatically powered up when the blood pressure portion ofthe History and Physical needs to be performed.

DICOM—includes the Digital Imaging and Communications in Medicineprotocol that provides a standard file format specifying how medicalimages are stored and presented along with information about the patientand the image such that any DICOM compliant viewer can present a DICOMcompliant image. It also defines the communication protocol (overTCP/IP) used to transmit the images

Attribute—data or functions associated with objects or records. In someimplementations, the attributes can be assigned or triggered throughselection (by, for example, either mouse click of or drag and drop of animage or diagram into an element in the active template.

Value—typically a number and its associated unit. In someimplementations, when the element is chosen, the attribute value thatwas encoded in the element is automatically associated with the imagewith the user and placed in the database as well as on the image asappropriate.

Context sensitive—for example, encoding the use of specific tools basedon information known to the integrated system, such as the modality, thebody part, the institution, patient demographic data including patientage and sex, the chosen element in an active template, the positionwithin an active template, or the chosen diagram or example image in thedecision-support system, or any combination of two or more of those.

Object-oriented programming—a method of programming in which each object(for example, a group of cohesive lines of code) has its functionsencapsulated wholly within the code. In some implementations, eachelement of the active template is encoded as an object which is assignedattributes that are related to the element which will be entered into adatabase (in the case of a data attribute) or a function such as aviewer tool choice which is displayed simply based on choosing theobject.

Objects—in some cases, objects are called elements in both the activetemplates and the decision-support system; the objects functionsimilarly to an object-oriented programming object in the sense that theattribute is essentially encapsulated within the element that can bechosen.

Guided mode—in some implementations, in this mode, the active template,for example, is presented essentially line by line to the user. In somecases, a beginner user (usually a person without much experience) isasked to evaluate an image for classification one criteria at a time.For example, the beginner can be asked to evaluate heart size on a chestx-ray, followed by evaluating the aortic configuration. In guided mode,in some cases, the active template presents one structure or finding tobe evaluated at a time while the decision-support system similarlypresents information relevant to correctly classifying the image relatedto the criteria being evaluated.

Expert mode—in some cases, in an expert mode, the expert user can viewthe entire active template at one time.

Feedback—for example, in a dual read mode feedback is an automaticresult when there is a discrepancy between a report (classification) ofa beginner user and a quality assurance user.

FIG. 1 shows a system 100 for users 105, 107, 109, to view, learn about,analyze, categorize, classify, or make decisions (or any combination ofthose) regarding structured content that is stored in and retrievablefrom one or more servers 102 through one or more user systems 104, 106,108, 110 connected to the server(s) 102 directly or through a network112. In some implementations, the content and functions of the systemare made available to the users by web servers associated with theservers 102 and web browsers running on the user systems. The usersystems can be workstations, notebooks, tablets, pads, mobile phones,and other fixed and portable devices. Examples of structured contentinclude medical images, e.g., diagnostic images, that are in compliancewith DICOM (Digital Imaging and Communications in Medicine). DICOM is astandard file format specifying how medical images are stored andpresented along with information about the patient and the image suchthat any DICOM compliant viewer can present a DICOM compliant image.DICOM also defines communication protocols, e.g., over TCP (TransmissionControl Protocol)/IP (Internet Protocol), for use in transmitting themedical images. Examples of diagnostic images include radiology images,e.g., including DX (Digital Radiography in plain films) images, US(Ultrasound) images, CT (X-ray Computed Tomography) images, MRI(Magnetic Resonance Imaging) images, nuclear medicine scan images, andmammograms. In some implementations, the structured contents are medicalcharts, e.g., standard History and Physical information, or non-medicalrelated information, e.g., accident investigation information, crimescene investigation information, restaurant menus, and interactiveeducation and testing scenarios. In some implementations, the structuredcontents are stored in one or more databases 114 that is part of theserver(s) 102 or connected to the server(s) 102.

The user systems 104-110 can be computer systems that include displaysfor displaying information to the users. In some implementations, thecomputer systems include input devices, e.g., keyboards, for the usersto provide input into the user systems. The computer systems may alsoinclude a touch screen through which the user inputs information to thecomputer systems. In some implementations, the user systems 104-110 arehandheld devices or mobile devices, e.g., a pad, tablet, or a smartphone. The network 112 can be Internet or other networks, e.g., LAN(local area network), VPN (virtual private network), or WAN (wide areanetwork). Example users of the system 100 include individuals, such asmedical professionals of different roles including doctors, nurses, andothers, or system administrators, and entities, such as hospitals.

The server(s) 102 can store and support one or more viewers 116, one ormore active templates 118, and one or more decision support systems 120,each of which can be displayed to a user through an interface to allowthe user to act on the structured contents stored in the database(s)114. In some implementations, the viewer(s) 116, the decision supportsystem(s) 120 and the active template(s) 118 reside on the same domain,which is the recognizable name associated with an IP address for aninternet resource, e.g., a server on the public internet, so that theymay interact with each other with a unique level of integration. Atleast part of the viewer(s), the template(s), and the decision supportsystem(s) is embodied in computer programs.

An interface can be understood as visual presentation of information ona screen with which a user interacts to perform functions related tohis/her role. In the example of the system 100, the interfaces for theviewer(s), the active template(s), and the decision support system(s)are integrated such that, change of the content in one interface can beautomatically populated into the other interfaces such that the contentsof the other interfaces are consistently updated. The interfaces canallow a user to view and classify structured contents into structuredreports.

As shown in FIG. 2, the interfaces 202, 204, 206 of the viewer(s) 116,the decision support system(s) 120, and the active template(s) 118 canbe displayed on a screen 200 of a user system simultaneously. Sometimesinstead of three interfaces, a user may choose to display only oneinterface or two of the three interfaces simultaneously. Sometimes thethree interfaces can be displayed on two or more screens, e.g., screensthat are arranged side by side. The three interfaces can be displayed indifferent left-center-right orders than that shown in FIG. 2. An exampleof the display is shown in FIG. 2A. A viewer interface 228 is displayedon a screen 222, and an active template interface 226 and a decisionsupport system interface 224 are displayed on another screen 224arranged left to the screen 222. The screens 222 and 224 have differentsizes. However, they can also have the same size.

In some implementations, displayed content(s) of one interface can bedragged and dropped into another interface, which can automaticallyupdate its contents based on the dropped content(s). In some cases, thefeatures of two or all three of the interfaces can be combined andpresented in a single interface to the user. In such cases, the featuresthat we describe here can still be presented, but they need not besegregated formally among two or three different interfaces. For examplea structured report can summarize the state(s) of the subject andcontain links to any or all of the three primary interfaces used ingenerating the structured report.

The viewer interface 202 can be a browser window which functions topresent visual representations (for example images) of the structuredcontent to the user for the user to view, understand, and act on. Theviewer system 116 is engineered to present DICOM compliant image data208 (including images and related data) along with tools 210 through theviewer interface 202 to a user. The viewer system 116 can servewebpages, deliver images, and interact in other ways with the browserrunning on the user system in order to enable the viewer interface toprovide the features that we describe, and others. The browser (forexample a Google Chrome browser) can take advantage of HTML5 featuresincluding the recent “canvas” element, for example, to quickly downloadthe image data (that represents the image) and store the image data inthe user system running the browser. One advantage of downloading andstoring the data on the user system is that the downloaded image datacan be manipulated at a higher speed locally than could the image dataif it were only remotely stored at the database(s) 114. In other words,the browser can manipulate the display of the image quickly for the userand therefore make the user experience satisfying.

The tools 210 enable the user to manipulate, change, analyze, measure,alter, and otherwise work with the image data and the image. Each of thetools can be context sensitive so that for different image data 208,different image capture modalities, different subjects, and differentactive template data, different tools are presented or the same toolsmay be presented differently with different features. Using the tools, auser can manipulate the image data 208, e.g., obtain variousvisualizations of image findings and/or make measurements of sizes orother characteristics related to the images, e.g., angles, attenuation,volume, etc.

The decision support system 120 runs on the server usingdecision-support data in the database, can present information relatedto the decision-support data through the browser of the user system, andcan receive information from the browser of the user system, manipulate,and store it in the database for later use. The decision-support systemcan aid a user to make decisions or medical findings about (for example,to classify structured data about) a medical state of a subject whosemedical data is studied through the viewer 116. The decision supportsystem 120 includes a knowledge base that contains both static knowledgeand dynamic data. The knowledgebase can reflect the knowledge of subjectmatter experts (SMEs) about medical conditions of subjects. Thedecision-support system interface 204 displays selected data and imagesfrom the decision-support system database, e.g., upon a user's choice,to the user.

The static knowledge is stored in the system 120 when the system isinitially constructed or from time to time by updating the system.Examples of static knowledge include knowledge related to differentsubject matters, e.g., anatomy, normal and abnormal findings, normalranges related to measurements, age of subject, position of findingwithin an anatomic structure, etc. Dynamic data is data added to thesystem 120 during the use of the decision support system 120. Forexample, data can be added into the system 120 based on images, e.g.,anonymized images, that represent examples of normal and abnormalfindings identified by the users. Generally, the knowledge stored in thedecision support system 120 is input or confirmed by expert(s) inrelated field(s), and can help users of the system 120 to perform tasks,such as classification, using best practices.

In some implementations, the knowledgebase includes known ranges thatare incorporated in element(s) of any one or more of the threeinterfaces. For example, specific pathologies may correspond to a knownage range or known gender. The systems can automatically check theranges against a subject's information to determine whether a user madea wrong classification or automatically exclude certain classifications.In an example of an active template having ranges incorporated, atemplate choice element, which can be in the form of an object, can havean age range attribute that limits the application of the templatechoice to only those subjects whose ages are within the incorporatedrange of the pathology. A user would be warned if he/she chooses afracture only seen in pediatric patients when the patient is over 18years old, when the age range is incorporated in the active template andassociated with the fracture.

In an example of incorporating ranges in a decision support system,image examples, e.g., in the form of objects, representing a pediatriconly fracture can incorporate an age range attribute such that if a userdrags any of the image examples into the active template for a patientolder than 18, the systems warns the user against using that specificexample in that patient's report.

In an example of incorporating ranges in a viewer, a “Kline line”measurement tool can have age ranges incorporated for identifying apediatric Slipped Capital Femoral Epiphysis (SCFE). If a user tries touse the tool on a non-pediatric hip, the systems warn the user of theimproper use of the tool.

Information in addition to ranges, such as gender, ethnicity, etc. canbe similarly incorporated to assist the user to make classificationsusing best practice. Such information can be incorporated in allsystems/interfaces and all the interfaces are integrated to allow bestpractice.

The active template interface 206 can enable the user to use template(s)maintained and served by the active template system 118 to make choicesand selections of elements of uniform descriptive information that areintended to be representative of a determined state of a subject. Acomplete set of choices and selections that has been made using anactive template can be considered a classification of the state of thesubject (for example, a diagnosis in the case of a medical subject). Inthat sense, the person who is the user makes use of the active templatein operating as a human classifier. Once a complete set of choices andselections has been made using an active template, the data thatrepresents the choices and selections can be stored locally and at theserver and used for a wide variety of purposes and distributed to a widevariety of recipients. For example, using a classic approach, thechoices and selections can be automatically expressed as a medicalreport on the medical state(s) of patients. The report could bedistributed electronically or on paper and used in a conventional way.In another example, the systems can automatically classify a medicalstate of a subject to conform to statutes and other classifications,such as ICD-9 and ICD-10. The medical report can automatically containthe standardized/statute-compliant classifications/coding. In someimplementations, the standardized classifications of the medical statesare enabled by associating those classifications as attributes to theactive template elements that denote their possible presence in thesubject. The association can allow the medical report generated based onthe active template to be automatically translated into the standardizedclassification/coding system. For example, sinus disease has an ICD-10code. This code is associated with the active template element(s) thatleads to the finding of sinus disease. As a result, a user's selectionof the template element(s) not only leads to the finding, but also thestandard coding of the finding.

The interface 206 can display an active template provided by the activetemplate system 118 as a number of dynamic series of display elements(such as words or numbers and alternative words and numbers) such thateach of the series potentially represents a set of syntactically correctsentences associated with an aspect or feature of the state of thesubject. The set of sentences represented by one of the dynamic seriescan describe the vast majority of (or all of) the findings that might bepresent on a specific study of the image or other structured content.Example of a specific study could include a modality body part relatedstudy, e.g., X-ray for chest, CT for abdomen, etc., for a given subject.

The active template system stores program objects that represent wordsor numbers that can be presented in display elements as parts of thedynamic series of display elements that go together to form thepotentially large set of syntactically correct sentences associated withvarious features of the state of the subject. The program objects storedby the active template system are based on knowledge imparted to thesystem by one or more subject matter experts. In other words, theprogram objects capture information from experts that can then bepresented through the active template and used by a non-expert user toclassify the subject and learn about appropriate classifications givenunderlying structured content about the subject and informationpresented in the decision support system.

In the example of a radiology study, the template system 118 cantherefore include what amounts to a routine checklist or way ofanalyzing the subject of a SME, e.g., a trained radiologist. Thetemplate system 118 incorporates the knowledge of a subject matterexpert and allows the active template to incorporate best practices inuse of terminology, order of report findings, appropriate descriptors,and accurate classification. A user can activate and use the expertknowledge through the template interface 206 simply by interacting withthe display elements. Interacting with the display elements can includeclicking (or image drag and drop) on them making choices amongalternatives, viewing tips, and observing the expansion and contractionof the number of elements included in the sentence comprising theelements, among other things. The template served by the template system118 can also allow a user to readily choose and use complex tools in theviewer interface served by the viewing system 116 to make findings(choices, selections, Classifications) with the assistance of thedecision support system(s) 120 and the template(s) 118. As a result, agenerally trained medical professional who is not specialized in thesubject matter of a particular study can produce an expert level reportusing the system 100. In some cases, even a relatively untrained personcan use the system 100 to produce an initial reasonably goodclassification and related report.

The system 100 not only allows the user to classify the state of thesubject accurately, quickly, and easily, it also provides a mechanismfor acquiring, storing, and imparting to the user knowledge of a varietyof subject matter experts. The knowledge is imparted in a simple, easyto understand, and smoothly integrated way that enables the user to takeadvantage of it, and to learn during the process of classifying thestates of specific subjects. The expert knowledge can be presentedthrough the decision support system interface and the active templateinterface, which both can work in an integrated way with each other andwith the viewing interface. In the course of making selections, andchoices in the active template interface, the user is prompted toconsider information that represents knowledge of subject matter expertsand is selected and presented in a way that is relevant to the contextin which the user is working.

In addition, the user can learn and can be managed based on specificfeedback provided with respect to classification work on the states ofindividual subjects. The feedback can include feedback that is providedautomatically by the system based on data about similar subjects,similar states, and expert knowledge. The feedback can also includespecific non-automated feedback provided case-by-case. For example, anexpert can review a report produced by the user and provide direct orindirect feedback indicating where errors were made, for example.

The feedback can be automatic feedback. For example, when there is adiscrepancy between the reports of the expert and the beginner, thebeginner can review a redline version of the report, in which where theexpert report differed from the beginner report has been automaticallynoted. This review can be prompted in real time once the expert is donewith evaluating the subject. The systems can also automatically generatean excellence score for the beginner by comparing the reports of theexpert and the beginner and adding up the level of mistakes related tothe value of the attributes. The mistakes can include both falsepositives and false negatives.

In addition, parties who have access to information that establishes theactual state of the subject can provide that information to the user,sometimes after the fact. For example, a surgeon who removed a canceroussection of a colon from a patient and who observed the actual extent,severity, and character of the malignancy can provide direct feedbackthrough the active template system to the user indicating the actualstate of the subject. In addition, this actual state information can beused in connection with the structured content as a basis for improvingthe underlying data stored in the system. For example, information ofthe actual states can be loaded into the systems automatically toimprove the data in the systems so that veracity of specific pathologiesis effectively crowd sourced. In some implementations, the informationof the actual states, e.g., surgical/pathological data can beautomatically correlated to specific images/subjects, e.g., be noted asan attribute of an image, in the decision support system. Theinformation can then be used as a surgically/pathologically proven caseof a specific state of a later subject image/study.

In some implementations, a less skilled pathologist, for example,located in a part of the world where the general skill of pathologistsis lower than in another part of the world, can provide usefulclassifications and related reports about subjects, and then the reportscan be quickly reviewed, critiqued, adjusted, and issued by a moreskilled pathologist located in the other part of the world. A widevariety of other use cases are also possible.

EXAMPLES

When the system 100 of FIG. 1 is for use in medical diagnosis based onmedical images, the viewer is browser-based and DICOM-compliant andprogrammed to be native with respect to a given browser, such as GoogleChrome. In some implementations, the viewer uses features available inHTML5, such as the canvas features, to view diagnostic quality, DICOMcompliant images. Diagnostic quality images typically follow theguidelines offered by the American College of Radiology. The FederalDrug and Food Administration (FDA) regulates imaging devices and someimaging devices that produce diagnostic quality images are FDA approved(401k). The viewer makes use of the browser capabilities to store theimages on the local hard drive for rapid retrieval by the user.

When images of a subject are to be studied by the viewer, instead of theactual subject, the system 100 receives the diagnostic quality, DICOMcompliant images from various client sources and provides reports to theclient sources. The DICOM standard includes the method of sending imagesto the system 100. Generally, all imaging devices throughout all medicalspecialties, including radiology, pathology, ophthalmology, GI(Colonoscopy), etc., manufactured since 1990 have the capability or canbe enabled to have the capability to send DICOM images to a DICOMdestination over TCP/IP. A DICOM destination includes 3 pieces ofinformation: (1) the Application Entity Title, which typically has 16characters, (2) the IP address, which can be the standard 4 block IPaddress or a domain name, like dpr.lifetrackmedicalsystems.com, and (3)the port over which the information is sent, which typically is port104.

A new client of the system 100 can be readily added to the system 100for uploading images and access the system. For example, the system 100can provide an approved new client with the information (1)-(3) of aserver on which the system 100 is operated. The new client then canenter the three pieces of information through a DICOM destination dialogbox/interface of an imaging device that is connected to the internet.The device then can communicate with the server, which is a DICOMcompliant destination. The destination IP address (or the second pieceof information) is the internet (or LAN) location of the server whichreceives the DICOM images, which can be a server located in theSingapore Amazon Web Services (AWS) datacenter.

Other than the clients, users of the system 100 also include readers ofthe images who generate the reports for the clients. Eligible readersmay include radiologists or other medical professionals. The readers canreview the images using a processor/workstation local to the LAN basedonsite server of the system 100. The readers can also review off the WANbased server of the system 100, e.g., the Amazon Web Services (AWS)based server, e.g., through the Internet, or off a mirror server that isa mirror of the AWS WAN based server of the system 100. The mirrorserver can be set up at a location different from the AWS server of thesystem 100 and can allow faster image retrieval by onsite readers sinceit is on site and images are retrieved to local workstations over theoffice LAN. For example, the mirror server can be located on the LANwhere a readers' workstation exists and can synchronize with the serverof the system 100.

The system 100 is highly de-centralizable and can receive input fromradiologists working from home with an inexpensive, low cost, e.g., lessthan $1,000, processor-based workstation. The workstation is capable ofacting as an “acceleration server” which pre-fetches studies assigned toa specific radiologist. The radiologists working for the system 100 canbe located anywhere in the world, e.g., anywhere that has consumerbandwidth (1-5 Mbps) connections to provide highly efficient radiologyreading services.

The system 100 can allow a client to store and archive images locally onthe LAN of the client. The client can host a server with the system 100at the location of the client, e.g., a hospital or an imaging center.For example, a client hospital or imaging center may want to have amirror of the entire system located on site in the event that internetaccess is unavailable intermittently. The server hosted by the clientcan also mirror all the same information on the server of the system100. The images in the system 100 can be archived both onsite on theclient LAN and on the server of the system 100, e.g., by using Amazon S3(Simple Storage Solution) hard drives. The client can access all studiesand reports even if the internet is temporarily unavailable to accessthe server of the system 100.

In some implementations, functionally, the client owns the images andthe system 100 owns the active templates and the decision supportsystem(s). The system 100 can also own all data related to the activetemplates and the decision support system(s), and can contractually ownthe anonymized images that are added to the decision support system(s).

In some implementations, referring to FIG. 3, when a user logs into thesystem 100 through a browser, a worklist 300 is displayed through aninterface 301 to the user working on a user system. The worklistprovides a list of records 302, 304, 306 for the user to view. Users whohave may have different roles with respect to subjects, e.g., nurses,doctors, or administrators, or users of different entities, e.g.,hospitals, have different permissions to see different records. Thelevels of permissions are encoded by the system 100 in association withthe user's registration to the system 100. The display and access of therecords are in compliance with the HIPAA Privacy Rules. From the list ofrecords, the user can choose one or more to review, alter, critique,conduct a study, or perform other functions. In this example, the user'suser name 342 and role 340 are listed in the interface showing theworklist 300. Additionally, the institution 344 to which the userbelongs is also listed.

In this example, the worklist 300 shows three records 302, 304, 306 inthe form of a table. The table has multiple fields that differentiateand/or summarize certain aspects of the different records. For example,the fields include patient name 308, patient ID 310, accession no. 312,institution name 314, modality 316, body part 318, #SE (number of imageseries)/#I (number of total images in the study) 318, study date 322,received date 324, description 326, QA (quality assurance) status 328,and status 330.

The patient name and the patient ID identify the subject of the medicalrecord. The patient ID is a unique ID for each patient within the system100. The accession number is a unique number associated with a studyordered at an imaging center. The institution 314 specifies the originof the medical records. The modality 316 is the type of image capturingtechnique used, such as magnetic resonance imaging (MRI). The body part318 identifies the part of body for which medical images are recorded.The SE 320 is the number of series, the #I is the number of total numberof images in the study. The study date 322 shows the time at which therecords are studied. The received date 324 records the time at which therecords are received by the system 100. The description 326 provides abrief description of the records.

The status 330 shows the status of the report to be generated orgenerated from the study of the corresponding record. The status can be“new,” “submitted,” “verified,” “dictated,” “preliminary,” and “final.”The status of “new” means that the corresponding record or case has notbeen reviewed or reviewed all the way through to generate a report. Thestatus of “submitted” means that the client who sends images to thesystem 100 has determined that the record is ready for presentation to areviewer, e.g., a radiologist. The status of “verified” means that therecord indeed has been verified, e.g., by nurses of the system 100, thata record contains all appropriate information for the initial reviewerto review. The status of “dictated” means that the initial reviewer hasmade their initial classification/report. The status of “preliminary”means that a second classifier/reviewer/advanced reader has reviewed therecord and the initial report generated by the initial reviewer and hasmade changes to the initial report. The status of “final” means that thereport is signed and placed by the classifier/reviewer/radiologist ofrecord.

The attributes of the status 330 can be named differently from the abovediscussed names. In addition, there can be other statuses, includingmedical statuses, financial statuses, or procedural statuses ofpatients. The worklist 300 can have fields related to a radiologyinformation system (RIS). For example, the worklist 300 can includeadditional fields for patient scheduling and billing. The worklist 300can also include derived fields, e.g., fields automatically generated byan active template, such as ICD-9/10 codes, conspicuity and severityvalues, and intelligent user efficiency related time stamps. Specificexamples of derived fields can include time of image receipt to time offinal report (i.e., turnaround time), and attributes related to thereports, e.g., the ICD-9/10 codes and CPT (Current ProceduralTerminology) codes. Generally, the CPT codes are related to an actualimaging procedure which can be determined by a radiologist noting whichviews and series have been performed. For example, a two view chest hasa different CPT code than a single PA view. The system 100 canautomatically determine the CPT code, e.g., based on the selection madeat the active templates. For these examples, the worklist 300 caninclude statuses of the patient scheduling and billing, the turnaroundtime, and the attributes related to the reports.

The RIS contains information related to patient demographics, schedulingof studies and billing, and patient tracking, e.g., when a patientarrives in the imaging department, when the patient leaves, whether itis inpatient or outpatient, information related to report distribution,including primary care physician, ordering physician, consultingphysician(s), referring physician, primary reading physician, secondreader, CPT and ICD-9/10 codes, etc.

The QA status 328 is a custom, non-DICOM field for noting if there arequality assurance issues, e.g., with the records received from theclient. The QA status may indicate that such issues exist and the system100 or a user of the system 100 may request the client to address theissues in order for the user or other users to process the record andgenerate a report based on the record. For example, if the client, e.g.,an imaging site, informs the system 100 that two views of a chest havebeen sent to the system 100 for an X-ray study, while the system 100only receives one view, the system 100 or a user of the system 100places a note for the missing view in the QA status 328. The note in theQA status 328 can be in the form of a QA code for easy searching. Also,the placement of the note sets the status 330 in a “New” status andnotifies the client to address the QA issue. In the example discussedabove, the client can address the QA issue by sending the missing or bychanging information about the number of views, e.g., headerinformation, to match the available views.

In some implementations, an authorized user may add additional fields tothe table. Sometimes a user is not interested seeing in all fields andcan click on a hide button 380, e.g., to hide the QA status field 328.In some implementations, the system 100 allows a user to show or hideany one or more fields so that the user can define his/her ownworklists. The user can perform functions on any of the shown fieldsand/or filtered fields, which can include any attributes of the fields,including those derived from the active templates, e.g., the ICD codes.Field filtering is described further below.

When the worklist 300 contains multiple pages of records, a user can usethe button 392 or the button 398 to reach the first or last page of therecords, or can turn pages of the record back and forth using buttons394, 396.

The system 100 allows a user to invoke a field filter on the worklist300 to build customized worklists from the worklist 300 based on theuser's needs and perform functions on the built worklists. The recordsin the worklist 300 can be sorted or filtered based on the contents ofthe fields. The sorted or filtered records may provide a user withinformation, e.g., statistical information, about the records orpatients. Each field of the record includes an activatable button 350that shows a sorting standard, e.g., alphabetical sorting from A to Z orfrom Z to A. Some activatable buttons for some fields can have moreadvanced sorting standards. For example, for the institution name field314, in addition to the alphabetical sorting, an additional filter 352is provided, e.g., popped up in the interface. Referring to FIG. 3A, asan example, the pop-up window 354 contains selectable phrases 356, 358,360, 362, 364, 366 and an open field 368 for a user to fill in. Thecombination of the selected one or more phrases and the content filledin the field 368 can form a filter string. The user can choose to applythe filter string by activating an apply button 370 or delete the filterstring by activating a delete button 372. The pop-up 354 can be movedwithin the interface 301 and can be closed by activating a close button390 on the pop-up. In addition, a user can search the records byentering search strings in a search field 382. In some situations,search is functionally a form of filtering supplanted by the fieldfilter functionality.

Filtering can be performed on any field of the worklist 300, includingthose fields that are hidden or derived, and those fields that areautomatically encoded by one or more active templates, e.g., thequality/conspicuity data encoded as attributes in the elements of thereports generated by the active templates, and the ICD-9/10 and CPTcodes. By defining the fields of a worklist, including the hiddenfields, and then filtering on these fields, including the derivedfields, a user can generate a worklist based on his/her needs. Forexample, a worklist that includes only MRI brain studies greater thanone hour old and received from a specific institution can be generatedfrom a general worklist of the system 100 that includes much moreinformation. The user can perform one or more functions on the generatedworklist by invoking the field filter function. For example, using thefield filter function, a user can find out which cases/records have notbeen read in a timely manner and can notify a hospital administrator anda radiologist to read these delayed cases/records.

In some implementations, the field filter function capability of thesystem 100 can be viewed as a form of language when used in conjunctionwith the general worklist, e.g., the worklist 300. The application offilters on worklist fields can create highly customized worklists (whichcan be viewed as nouns) upon which specific functions (which can beviewed as verbs) can be performed. The application of a function in aworklist is analogous to applying a verb to a noun to create unique andpowerful combinations intuitively. Applying filters on fields createsunique worklists which can be viewed as the equivalent of nouns. Thislanguage can provide a powerful, intuitive paradigm in which filtersacting on fields to create custom worklists “nouns” in an intuitivemanner. Specific functions (verbs) can then act on these customworklists to cause powerful and intuitive actions to occur on specificsubsets of the worklist information. For example, in the institutionname filter shown in FIG. 3A, a user can filter on a field usingfunctions including “is,” “is not,” “contains,” “does not contain,”“begins with,” “does not begin with,” etc. (not all shown). In theexample of time fields in the worklist, the filters can be applied todates and/or times, e.g., “is less than” or “is greater than” “Xhour(s)/days/weeks/months/years,” “is done” or “is not done” on specificdate(s), date range(s), etc. Application of the field filters cangenerate useful and highly specific worklists based on the needs ofdifferent users.

The users and/or the system 100 can then perform functions on thegenerated worklists. Examples of the functions can include notificationsto hospitals, radiologists, or other individuals/entities using, e.g.,short message service (SMS), email, phone call, etc.; and Excel/commadelimited export of the data in the generated worklists. Users, e.g.,administrators, can determine patient throughput from the exported data,and radiologists can determine their own efficiency and quality orsimilar data on others.

A user can choose one or more of the records for study/review byclicking on the patient name in the patient name field 308. Theactivation of the patient name causes a display of the medicalinformation stored for the patient. In some implementations, theactivation directly leads the user to a viewer interface in whichmedical images are displayed in a viewer. A user may also access aviewer interface by clicking on a viewer button 384 on the interface301. The activation of the viewer button 384 opens one or more viewerinterfaces that include studies corresponding to those records havingthe check boxes in the leftmost of each record checked. One or morestudies can be simultaneously opened. In addition, medical reportsgenerated for the records can be viewed by activating the status of thereport in the status field 330. A user can further access activetemplates from the report(s) (see, description of FIG. 4 below). Asdescribed previously, the template(s) and the viewer(s) can besimultaneously displayed to the user through multiple interfaces.

The values shown in the cells of the table are populated frominformation stored at the server in the database.

Referring to FIG. 4, a viewer 400 is displayed through a viewerinterface 402 presented by a browser to a user when the user chooses tostudy a record of the worklist 300 of FIG. 3. In this example, the userclicked on the patient name of the record 306, which contains medicalrecords of MRI performed on the brain of the patient. The viewer 400includes a record region 418 in which medical images, e.g., MRI imagesof the head, are displayed in a thumbnail form. The viewer 400 alsoincludes a working field 420 into which one or more of the MRI images inthe region 418 can be dragged and dropped for detailed analysis.Although four images are shown in the working field 420, when the vieweris open, the working field is empty and the images can be added orremoved from the field 420. In addition, the viewer 400 includes a set422 of tools for use in analyzing the images.

When the record is first-time selected, the selection initiates adownloading process for the MRI images 404, 406, 408, 410 and allavailable tools from the server/database of the system 100 of FIG. 1 tothe local hard drive of a user system on which the viewer interface 402is run. A viewer code base associated with the downloaded images and thetools can be stored in the browser cache. As a result, when a user makessubsequent accesses to the viewer 400, the downloading does not have tobe repeated. When one or more features of the viewer or the system 100are updated, a user can clear the browser cache and reload the browserto effectively upgrade the viewer and the system 100 accessed by theuser through the browser.

The progress of the downloading are shown to the user in a progresswindow 412, which includes one image bar 414 and one total progress bar416. During the downloading, the bars are dynamically filled to indicatethe downloading progress. The image progress bar 412 shows progress ofdownloading the current image only, whereas the total progress bar showsthe progress of downloading the entire study. In addition, thedownloading progress of each individual image is shown in acorresponding bar 424 under each thumbnail image. The bar 424 containstwo numbers, one in front of a slash and one behind of the slash. Thenumber in front of the slash shows the number of images available on thelocal hard drive of the user out of the total number in the seriesdenoted by the thumbnail. The number behind the slash indicates thetotal number in the series represented in the corresponding thumbnail.The number in front of the slash increases from 0 during the downloadingand when the downloading is completed, the number equals the numberbehind the slash, thus identifying when all the images in the serieshave been loaded to the local hard drive. In some implementations, thecolor of the bars 414, 416, 424 can turn green to indicate completion ofthe downloading. The viewer 400 is configured such that the downloadingprocess takes only a short period of time, e.g., less than 10 seconds,less than 5 seconds, or less than 1 second. Different images and toolscan be downloaded simultaneously.

A user can use the tools in the tool set 422 to manipulate the images inthe working field 420 to measure characteristics of the images and toannotate the images and their characteristics. Example manipulationtools include but are not limited to: magnification tool 424, pan tool426, window level tool 428, rotate tools 430, 432, flip tools 434, 436,scroll or browse tool 438 for scrolling or browsing through a stack ofimages, e.g., in CT or MRI, invert color tool 456, view point selectiontool 458, full screen tool 460, and drag and drop tool 462. Examplemeasurement tools include but are not limited to: linear distance/sizetool 440, volume tool (not shown), and Hounsfield units tool 452, e.g.,for measurement of x-ray attenuation, which can roughly be electrondensity of matter, and angle(s) 442, 444. The measurement tools can alsoinclude any number of context sensitive measurement tools that areautomatically chosen through the use of the decision support system(s)and the active template(s). For example, for MRI studies, spine discbulge measurement tool(s) can be automatically chosen and shown whenreading an MRI of the lumbar spine, or in a calcaneus fracture when theuser identifies a definite or possible calcaneus fracture the Boehler'sangle measurement tool(s) can be automatically chosen and shown. Exampleannotation tools include but are not limited to: text tool 446, labeltool 448, removal tool 450, and annotation manipulation tool 454. Inaddition, the tool set 422 also includes a button 464 that allows a userto access the report editor and active template(s), a button 466 toaccess the decision support system(s), and an information tool button467, the activation of which shows a user the DICOM header informationof the study.

The tools are shown in the interface 401 in the form of icons thatgraphically explain the use of the tools to aid a user in using thetools. To facilitate the usage, when a user moves a mouse or figure ontoan icon, a brief explanation of the icon and the tool appears on theinterface. Activation of a selected tool allows the user to apply thetool in the viewer 400. Additionally, a tool can also function as anobject that has attributes, which includes showing an appropriateportion of the decision support system to provide a user with detailedinformation on the use of the tool. For example, the detailedinformation includes the significance of the tool in the classificationof the subject under examination. The attributes of the tool object canalso be related to information contained in the study, e.g., the patientage and gender, such that a user using a tool that measures the volumeof the ovaries to study a male patient is warned.

The tools shown in the tool set 422 with the images for a particularstudy can be sensitive to the context of the study. In someimplementations, the use of specific tools is encoded in the tools basedon information known to the system 100. The information can includecontents of the images, and/or contents of the other interfaces,including interfaces displaying the active template(s) and the decisionsupport system(s). For example, the information includes the modality,and/or the body part, and/or the client, and/or referring physician,and/or reading physician of the images to be studied. In someimplementations, the tools are related to one or more activeinteractions the user made with the active template(s) and/or thedecision support system. For example, specific selections of choices ormodifiers from the active template(s) and selection of diagrams, images,or text elements from the decision support system can affect the tools.The details of these templates are discussed further below.

In some implementations, a user can encode the tools, and by encodingthe tools, a user can use specific tools as desired. For example, theclient is a podiatrist and may want all of his/her foot films to havemultiple specific measurement tools used on all foot radiographicstudies regardless of whether there is a determination of normal orabnormal. A tool can also be encoded automatically by the system 100 tohelp discourage inappropriate use of the tools and encourage appropriateuse of the tools based on the attributes of the tool. For example, atool may be encoded such that the tool is to be used on calcaneus only,or is to always measure a volume of the ovaries on a pelvic ultrasoundof a female. In addition, a tool can be encoded such that the tool istriggered by clicking on an image or diagram from the decision supportsystem. For example, the image of the decision support system can beassociated with the use of a specific tool such as Boehler's angle on acalcaneus fracture or Kline's line on a pediatric pelvis being evaluatedfor SCFE (slipped capital femoral epiphysis).

As a result, the integration of the RIS (Radiology Information System)with its knowledge of the patient and the images, in conjunction withthe knowledge of the chosen elements in the active template(s), thesystem 100 can automatically inform a user which tool to use to make aspecific measurement for the chosen elements. For example, the system100 can highlight the tool in the toolbar and/or a popup can begenerated on the viewer, if appropriate. In some implementations, when auser chooses a tool for use, the tool has been automatically encodedwith the desired measurements or manipulation the user would need toapply. For example, in a calcaneus fracture, the Boehler's angle tool isautomatically activated on noting the presence of a calcaneus fracturewhen an appropriate element in the active template is selected via mouseclick or drag and drop of the image showing the fracture. With theBoehler's angle tool being active, a first user click on the image atthe appropriate location determines an initial point of measurement ofthe angle. A popup can guide the user to perform the rest of themeasurements. Alternatively, the system 100 can guide the user by havingthe appropriate information displayed on the decision support system.

In some implementations, a full set of tools in the system 100 may notneed to be shown for particular studies, and different tool sets havingdifferent numbers or types of tools can be shown for different studies.For example, the Hounsfield tool used for a CT study is not shown foruse on an MRI study. By not showing unnecessary tools based on contentknown to the system, the viewer interface is cleaner.

In some implementations, the tools contain normal values that correspondto the context of the study. For example, a tool for use in measuringthe acromioclavicular distance can have a normal distance smaller than 6mm as an attribute of the tool. If the measurement performed by the toolshows that the actual distance is larger than the normal distance, theuser can be informed of the abnormality of the measurement, e.g., in apopup and. In some implementations, the system 100 can automaticallypreselect the active template elements that describe the abnormality.Additionally, a tool may have an attribute that adds the appropriateactive template element that describes a measurement result by the tool,and/or that suggests the user of other tools. For example, the templateelement to be added includes acromioclavicular separation types, and theother tools include acromio-humeral distance measurement tools.

A measurement tool for measuring a feature in a particular study can beencoded with a value or a value range in which the feature is deemed asbeing normal and/or other functions as attributes of the tool. Ameasured value different from or outside of the normal values/normalvalue ranges encoded in the measurement tool can indicate that thefeature is an abnormal medical state. Furthermore, the button 466 canguide a user to the relevant decision support system page describing thepotential abnormal state to obtain examples of normal or abnormalvalues, images, and diagrams to facilitate the use of the tools. Theuser can be prompted to use other related tools to evaluate commonlyassociated abnormalities.

In some implementations, the system 100 automatically associates valuesobtained from use of the tools, e.g., the measured value(s), with theactive template(s) of the same study and/or a report generated from theactive template(s). The automatic association can be triggered byactivation of a tool itself, or by a tool's context sensitive use in thestudy. For example, the system 100 can insert the values into structuredreports when the drag and drop tool 462 is used to drag and drop animage with a measurement or annotation into an active template in atemplate interface displayed simultaneously to the interface 402.

In some implementations, when an image is moved from the thumbnailregion 410 to the working region 420, the image is automaticallyannotated based on stored information about the image. The storedinformation can include information received with the records, e.g.,from different institutions or individuals, information entered byadministrator of the system 100, and information from the past use ofthe active template(s) and/or the decision support system(s). In theexample shown in FIG. 4, the four images 404-410 in the working region420 are annotated in text on a side of each image. The text lists thestudy description, study date, institution, physician, patientinformation including patient ID, name, date of birth, age, and sex. Inaddition, information about the image, including the modality, the bodypart, the series number, the instance number and the plane, is alsolisted. In addition, the plane in which the image is shown isautomatically labeled. For example, the images 406, 408, 410 are in theaxial plane and the four directions A-front, P-back, R-right, and L-leftare labeled. In another example, the image 404 is in the sagittal planeand the four directions H-Head, F-Foot, A-anterior, and P-posterior arelabeled. During a study, further use of the tools in the viewer, theactive template(s) and/or the decision support system(s) canautomatically and dynamically update the annotation. For example, when adistance between two points on the head is measured, description of thedistance and its measured value can be listed.

As another example of the system 100 using its knowledge about the caseto prompt a user to take the right steps in classification, when theuser is reviewing a particular part of the body of a subject, e.g., theright shoulder, template elements for the left shoulder are grayed outin the active template so that the user does not mistakenly classify theleft shoulder based on review of the right shoulder.

Example Active Templates

The active templates can facilitate a user to build a comprehensivemedical finding report based on information presented in the viewer.Using the template(s), a user can diagnose a medical state of a patientin connection with the patient's record shown in the viewer, ordetermining/classifying the data of the record relative tobaseline/normal ranges based on prior knowledge/aggregated data. Thesystem 100 stores multiple model templates for use in classifyingdifferent studies. When a study based on a record starts, a user canaccess a database that stores the model templates and choose anappropriate one based on the context of the study, e.g., the content ofimages in the record. In some implementations, the system 100 uses theknown modality and body part being studied in the viewer, which complieswith the DICOM standard, to automatically load the correct activetemplate, if it exists. When a study has been partially conducted orcompleted, an active template for this particular study has been savedin the system 100. When a user accesses the template to continue withthe study, instead of the model template, the previously saved activetemplate for this particular study is loaded and displayed to the userbased on the system's knowledge of the study.

The user can access the database storing the active templates frommultiple interfaces, e.g., the status column 330 of the worklist 300 inthe interface 301 of FIG. 3 or the button 464 of the tool set 422 in theviewer 400 shown in the interface 402 of FIG. 4. In someimplementations, a selection interface is displayed to a user, e.g.,simultaneously with the display of one or more of the worklist interface301, the viewer interface 402, and a decision support system interface(discussed below), to allow the user to choose a desired model template.In some implementations, when a study has been completed and verified, auser may directly access the report for the study, through which theuser can access the active template and then generate the report.

An example template selection interface 600 is shown in FIG. 6. In awindow 602, a user can view available template choices by clicking on abutton 604, which shows a drag-down menu with a list of availabletemplates. FIG. 6A shows an example menu 606. In this menu, thetemplates are named with three alphabetical strings 608, 610, 612 and anumerical string 614. In this example, the first alphabetical string 608is a two letter code that complies with the DICOM standard formodality/imaging device. The code denotes the medical methods used inobtaining the medical records or images. For example, “CT” represents CTscan, “DX” represents diagnostic X-ray, “MR” represents MRI scan, and“US” represents ultrasound. The second alphabetical string 610 denotesthe body parts on which the medical methods are applied. For example,“ABD” represents abdomen, “ANE” represents ankle either, “CHT”represents chest, and “CSP” represents cervical spine. The thirdalphabetical string 612 denotes the client/institution/generic template.For example, “HWY” represents Healthway, and “DEF” represents default.The numerical string 614 denotes any of 1000 or more variations of thetemplates to allow for potential variations based on referringphysician, reading physician, and any other variable that may determinethe most optimal active template. For example, “001” represents thefirst version of the active template and “002” represents the secondversion of the active template. Although the template choices arearranged in alphabetical order and the specific strings are used to nameeach template, the template choices can also be organized in differentorders and different names can be used for the templates. For example,additional blocks, e.g., three letter blocks, can be added to thetemplate names to represent language, country, physician, CPT (i.e.,number of views), and etc. The templates can be named in as complex away as a user might want.

The template selection interface can have other forms different from theinterface 600 of FIG. 6. Another example template selection interface660 is shown in FIG. 6C. The interface 660 is similar to the interface600 and allows a user to select a desired active template from allactive templates available in the system 100 in a window 662. Comparedto the interface 600, the interface 660 contains additional informationincluding the DICOM header information 664 and the current status 666 ofthe case/record. A user can change the status to dictated or submittedusing buttons 672, 674 or save the status using a button 670. Inaddition, the interface 660 is incorporated in a report window 668, fromwhich the user can access, generate, or cancel a report.

Although the template selection interface can be used by a user whennecessary or when a user chooses to, the selection of the appropriatetemplate can be automatically performed by the system 100. For example,the system 100 can choose a template based on the DICOM headerinformation including the Modality, and the Body Part, the RIS which hasthe information on the Institution and User Login, etc. Thesystem-chosen template can be automatically loaded and presented to auser, while the user has the option of choosing a different templatethrough the template selection interface.

In some implementations, the system 100 stores a template for each knownmedical study of a body part so that a template is available for allconventional radiology studies of any body part (would likely use CPTcode (which tells how many views) to select). The system 100 canautomatically choose a template for a user to use, e.g., using the CPTcode. A user can choose a desired template, when the automaticallyselected template is inappropriate or the user prefers a differenttemplate, by choosing one of the template choices, and the choice isthen shown in a window 616. Information of the chosen template, such astemplate name 624, size, 626, and date modified 628, is displayed. Theuser can then load the chosen template by activating a “load” button 618and use the loaded template for generating a report. The user can alsoedit the chosen template by activating an “edit” button 620. Forexample, when a user finds that the desired template contains error(s)or needs to be supplemented with additional elements, the user canaccess the source code of the template to edit the template.

In situations the user cannot find a desired template, the user cancreate a new template by activating a “new” button 622 and enter thecontent of the new template in a template editor, an example of which isshown in FIG. 6B. In the example template editor 640 displayed in aninterface 642, a window 644 is available for a user to enter templatecontents, e.g., in the form of source code. The user can choose to save(646) the entry, save the entry with a new name (648), create (650)another new entry, or leave the interface 642 back to the selectioninterface 600 (652). In some implementations, the interface 642 alsodisplays a cheat sheet 654 containing notes, explanations, and exampleson how to enter the code properly in the window 644 in order to create anew template. A user can choose to save an incomplete template and workon creating the template at a different time.

The template editor can also have other forms. An example editor 680that has a different form than the editor 640 of FIG. 6B is shown inFIG. 6D. Compared to the editor 640, the editor 680 has a “SaveTemplate” button 682. Through the editor 640, an authorized user canname the edited template anything the user would prefer.

Referring again to FIG. 6, when a user chooses to edit an existingtemplate, which can be a complete template or an incomplete template, aninterface similar to the interface 642 of FIG. 6B displays anothertemplate editor similar to the template editor 640 of FIG. 6B, exceptthat in this other template editor, there is existing entry in thewindow 644. In some implementations, the different actions, includingloading the template, editing the template, and creating a new template,require different authorization levels. Only users having apredetermined level of authorization are allowed to perform a certainaction. For example, a user needs to have administrator's authorizationto edit or create a template, while any registered user may be able toload and use the template.

The selection interface 600 of FIG. 6 also includes a cheat sheet 660that provides instructions to a user for using a loaded template ingenerating a report. Referring to FIG. 5, as an example, a loaded activetemplate 500 for X-ray of the chest performed at institution Healthwayversion one is displayed on a template interface 502. The activetemplate 500 can be in the text form and can include structured templateelements. Each template element includes one or more words or phrasesthat can describe the study. Some template elements contain alternativedescriptions of the study and a user can choose among such descriptions.The template elements can structured into categories including technique504, comparison 506, clinical history 508, impression 510, patientposition 512, mediastinum 514, etc. Each category can be labeled with aheader in bold. The header can be chosen to describe the category oftemplate elements. There can be other categories for this study. Thecategories are chosen such that each category describes the study underone criterion, e.g., medical method, patient information, etc. A cheatsheet 530, e.g., the same as the cheat sheet 660 of FIG. 6, is alsodisplayed in the interface 502 to help a user in choosing the templateelements.

The template elements are structured by subject matter experts who areknowledgeable about different areas of studies. Display of thestructured template(s), e.g., in the form of categories, can guide auser through a complex classification one step at a time. In addition,both diagrammatic and example based decision support by the decisionsupport system can be provided to the user in synchrony with thetemplate elements contained in the active template(s).

In some implementations, a template is displayed in a so-called guidedmode. In this mode, the active template is presented to a user line byline with all built-in template elements, such that a user withoutsubstantial experience, e.g., a beginner, can evaluate an image forclassification one category or criteria at a time. For example,following display of the template elements, a user first evaluates heartsize on a chest X-ray, and then the aortic configuration. In the guidedmode, the active template(s) present one structure or finding to beevaluated at a time. The decision support system can similarly presentinformation relevant to correctly classifying the image related to thecategory or criteria being evaluated. An example of a template 560displayed in the guided mode is shown in FIG. 5A. In the template 560,only one category 562 “Heart” and its template elements are displayed.Before a user selects the template elements in this category, no othercategories are shown.

Alternatively, a template is displayed to a user in a so-called expertmode. In this mode, all categories of the template are shown to theuser. The user applies his/her knowledge to build the report by notingwhat is abnormal or requires comment/editing. Generally, a user usingthe active template(s) in the expert mode is an experienced user. In theexpert mode, the user can view the entire active template all at thesame time. Based on the user's experience in classifying similar imageson the viewer, the user can jump directly to any point in the activepoint and select only those template elements relevant to classifyingthe pathology demonstrated on the image. Any one or more irrelevantcategories can simply be ignored. In some implementations, the user cansimply note directly that the image is normal and accept the defaultnormal values encoded in the active template. In this expert mode, theuser can also move to those areas of the template that need editing andexpand the selections to note the abnormalities by mouse selection ortouches. Alternatively, the user can also drag and drop the images beingstudied into the active template to automatically annotate the imagesbased on the location(s) at which the images are dropped into the activetemplate. Drag and drop is described below in more detail.

The template 500 of FIG. 5 is in the expert mode. Another example of atemplate 570 in the expert mode is shown in FIG. 5B. Compared to thetemplate 500, the template 570 is incorporated in a report window 574,which includes DICOM header information 576, current status 578, andfindings 580. From this window 574, a user can access the reportgenerated from the active template 570 and in the text form through abutton 582. For the purpose of discussion, the description below focuseson the template 500 of FIG. 5. The description is also applicable to thetemplate 570, unless otherwise specified.

A user of the template 500 can go through the template elements one byone and choose the applicable ones based on his/her study of the recordof the patient, e.g., the images shown in a viewer. For example, whenthe image studied shows a PA (posterior-anterior) view, the user choosesthe “PA view” template element 532. Each template element is can bechosen by a mouse click or a touch on the element one or more times orvia drag and drop of the appropriate image from the viewer.

A user can determine whether the text of the selected template elementsshould be displayed in the final text version of the report in bothfindings and impression, impression only, or findings only. For example,the user can determine by making different number of clicks on thetemplate elements. The template elements may change color to reflect thedetermination (described further below). In some implementations, the“Findings and Impression” sections in a report relate more to findingsbeing something observable, whereas the “Impressions” sections canconvey not only the finding, but also what the finding means related toa diagnosis. In each category of the active template, template elementsthat describe findings, impression, and both findings and impression arelisted for a user to choose. In some implementations, the selectedtemplate elements, e.g., for findings and/or impression, are displayedin both sections of the active template, while sometimes findings arereported only in the body of the report and impressions are only in theimpression of the report.

In some implementations, the template elements are shown in an initialcolor, e.g., grey, and each click or touch on the template elementschanges the shown color to a different color. As explained in the cheatsheet of 660 of FIG. 6, a template element can turn into three differentcolors from its original color when the user clicks on or touches thetemplate element for different numbers of times. For example, when auser clicks on an unselected template element once, the template elementturns green and becomes selected. This selection will incorporate thecontent of the template element in both the report and the impressionlist. The user can click on the selected, green-colored template elementagain (i.e., two clicks in total) to turn the template element into red.The so-chosen template element is then only included in the impressionlist, and not in the report. Finally, the user can click on theselected, red-colored template element (i.e., three clicks in total) toturn the template element into blue. The so-chosen template element isonly included in the report, and not in the impression list. If the userclicks on the blue-colored template element, the template elementreturns to the unselected status and becomes gray. Colors other thangrey, green, red, and/or blue can be used, each color can correspond todifferent choices of the template elements, and/or different clicknumbers can be associated with the different choices.

In some implementations, some of the template elements are associatedwith each other such that when a user selects one of the templateelements, the other associated template elements are automaticallyselected. For example, some template elements are adjectives ormodifiers that are frequently used to describe other template elementsthat are features of choices for a finding. For example, small roundmicrolobulated are adjectives commonly used to describe a lung nodule.The associated elements can also form an element cluster, e.g., commonlyassociated findings. Real time guidance can be triggered when oneelement of an element cluster is activated. Selection of adjectives of achoice can trigger automatic selection of the choice. For example, thetemplate element “AP” (anteroposterior) 534 and the template element“PA” 536 each is associated with the template element “view” 538 so thatwhen either the element 534 or the element 536 is selected, the element538 is automatically selected. Additionally, selection of someadjective/modifiers can trigger alternate choices. For example, when auser selects a modifier like “multiple,” an alternate choice whichincludes pluralization of the choice is automatically triggered.

In some implementations, some template elements are mutually exclusiveor internally inconsistent, such that choosing one precludes choosingthe other. For example, cardiomegaly and small heart are mutuallyexclusive. The active template may automatically gray out a templateelement that is exclusive to a chosen template element so that the userdoes not mistakenly choose exclusive/inconsistent template elements.

An active template can also have default selected elements. For example,“PA view” in the technique category 504 or “Normal” in the heartcategory 520 is a default selection. When a user selects anything in thesame category that is different from the default selection, the defaultselection is automatically deselected. The default selections can helpimprove a user's efficiency. Also, with the default selections, acomplete report can be generated to describe the medical status of thesubject in all aspects, without a user needing to choose typicallynormal features or needing to drop these normal features from thereport.

In some implementations, a user may want to describe a study or part ofa study, e.g., the technique, using template element(s) that is notlisted. The user can create additional template elements at the end of acorresponding category. In the example shown in FIG. 5, a user can clickon the dots 540 at the end of a category to open a fillable text box540. In the alternative template 570, instead of dots 540, the user canclick on pencils 572 located at a similar location as the dots 540 inthe template 500. The user can then enter desired words or phrases intothe text box 540 to form a new template element. The newly formedtemplate element can have the same features as the existing templateelements.

In some implementations, for a category, multiple of the same header maybe needed to describe or summarize a study. For example, a study maydemonstrate multiple lung nodules. Or the user may want to describe thesuperior mediastinum separately from the middle mediastinum. Thetemplate 500 shows only one header 513 for right lung nodules and oneheader 519 for the mediastinum. However, a user can expand a headerunder a category by clicking on an expansion sign (or a plus sign) 544located next to the header. In this example, by clicking on theexpansion sign 544 next to the mediastinum header, additional identicalheaders appear in the template. Multiple clicks can lead to addition ofmultiple additional headers.

In some implementations, a user may not remember or understand the termsin the template elements, making it hard for the user to make thecorrect choice. In other implementations, a user may not be experiencedin the field of the study, e.g., a medical student or trainee, or amedical professional whose specialty is in a different field. The usecan access a decision support system that helps the user understand theterms and/or the field of study. The user can access the decisionsupport system through multiple interfaces, e.g., the interface 402 ofFIG. 4 or the interface 502. On the interface 502, signs 548 atdifferent locations, e.g., near category names or within a templateelement, can be activated via mouse click(s) to access the decisionsupport system. In some implementations, when a sign 548 near a categoryname is activated, the user is directed to a part of the decisionsupport system associated with the corresponding category; when a sign548 within a template element is activated, the user is directed to apart of the decision support system associated with the content of thatparticular template element. In some implementations, a user can performcognitive matching or classifying of image data using the templateelements with the assistance of the decision support system. Details ofthe support system are described further below.

In some implementations, not all template elements need to be chosenmanually by a user. Instead, one or more, or many template elements canbe automatically chosen based on a user's interaction with the viewerand/or the decision support system. For example, a user may drag anddrop an image from a viewer, e.g., the viewer 400 of FIG. 4, onto acategory of the template, the correct template elements areautomatically chosen without needing the user to analyze the image ormanually make the choice. An example of the drag and drop process isshown in FIG. 5A, where a viewer interface 560 and an active templateinterface 562 are simultaneously displayed to a user for an X-ray studyof a patient's chest. The viewer interface 560 shows a viewer 564 thathas similar or the same features as the viewer 400 discussed withrespect to FIG. 4. The template interface 562 shows an active template566 that has similar or the same features as the active template 500 ofFIG. 5. A user can drag and drop (568) an image 570 in the thumbnailformat from the viewer 564 onto the technique category, e.g., on any ofthe selectable template element, the template elements “PA” and “view”automatically selected because the system 100 automatically determinesthe view of the image 570. Alternatively, a user may also drag and drop(572) the image in the working region of the viewer using the drag anddrop tool in the tool set.

When a user uses drag and drop of a PA chest image to select the “PA”template element, the system 100 obtains information about the image,which allows for the automatic selection of the template element. Theinformation includes that the image represents a specific view of thechest, i.e., a PA view and that the image is identified as a PA view forthe purposes of examples for beginners as to what PA views look like. Asa user reviews the case, additional information about the image isobtained, e.g., when the user identifies that there is cardiomegaly. Theadditional information is added, e.g., concatenated, to the existinginformation about the image known by the system 100. As a result, thesystem 100 can now automatically identify a PA chest which demonstratescardiomegaly.

In addition, when a user selects a modifier using drag and drop, thechoice associated with the modifier is automatically selected. In someimplementations, when a user drags and drops an example of complexpathology, which has a common best practices description from thedecision support system, the active template is automatically populatedwith the best practices with some of the elements “pre-selected” orautomatically selected. Choice of some active template elements maytrigger pre-selection or addition of template elements based on theattributes of the selected active template elements. For example,selecting a type I, II, III, IV, or V Rowe classification calcaneusfracture would automatically add appropriate best practices descriptionsand potentially automatically preselect the modifiers and choices asappropriate to describe that type fracture.

When a template element is selected, e.g., manually or automaticallythrough drag and drop, attributes, which are data or functions that havepreviously been assigned to the selected template element, aretriggered. Each template element (in the form of an object) can haveunlimited attributes. The tool objects of the viewer and objects in thedecision support system can have similar features. The attributes can bedata attributes, e.g., numerical, alphabetical, or both, which can befixed or variable, or function attributes, e.g., functions that cause atool in the viewer to be displayed and highlighted, or that cause an SMSto be sent. Examples of data attributes include ICD-9/10 codesassociated with the template element within the active template(s). Forexample, if a “Cholecystitis” related diagnosis is chosen in an activetemplate, the ICD-9/10 code would automatically be assigned to thecorresponding image(s)/record. Examples of function attributes includepresenting the Boehler's angle measurement tool in the viewer when auser selects a template element in the corresponding active templatenoting a calcaneus fracture. Another example of a function attribute isassociated with the function of inserting into an active templatediagram by drag and drop. For example, an acromioclavicular separationType II image can be dragged and dropped into an active template at theAC (acromioclavicular) joint element, which then can trigger theauto-annotation noting a Type II AC separation onto the image, andmeasurement tools to measure the AC separation distance can pop up inthe viewer displaying the image.

Each data attribute has a value, e.g., a number, and a unit associatedwith the value. The value and the unit can be encoded in theircorresponding template element. When a template element is chosen, thesystem 100 automatically associates the corresponding attribute valuewith the image/study or user, and places the attribute value into adatabase, e.g., the database 114 of FIG. 1, and on the image beingstudied, if appropriate. In some implementations, a subject matterexpert determines at the time of adding the attribute whether it isappropriate to place the value into the image, database, or both.Examples of values include ICD-9/10 codes related to imaging findingsand measurements, which can be inserted into the report and/or beauto-annotated onto the image and user mistake scoring. User mistakescoring is discussed further below.

In some implementations, not all template elements in a category aredirectly shown in the interface 502. Instead, the template elements canbe nested, e.g., under a “more” button 546, to keep the template shortand neat in its presentation. Sometimes an experienced user, e.g., amedical professional trained in the field of the study, may not need toreview all possible template choices and can make a diagnosis quickly bydirectly entering his/her diagnosis/comments in each category or inselected categories. The nested template elements can expedite theexperienced user's diagnosis process. Sometimes in a category, the mostfrequently chosen template elements are directly displayed while theother template elements are nested. Other times all elements in acategory are nested. When all elements of all categories of a templateare nested, the template is in a collapsed mode. The template 570 ofFIG. 5C is in a collapsed mode.

For the template 500 of FIG. 5 to be in a collapsed mode, instead ofonly the categories 518, 520 in FIG. 5, all categories of a template aredisplayed in the collapsed manner. Without expanding the categories todisplay the template elements, the user may directly enter “normal” or“abnormal” within each category or selected categories that arenecessary for the diagnosis, e.g., in the category 520.

In some implementations, through the guidance of the system 100, e.g.,links on various interfaces, a user of the expert mode can also gainaccess to diagrams and/or examples of normal and abnormal findingsrecorded in a decision support system. The user can choose a relevantmatch in either a diagram or an example. Based on the user's choice, thesystem 100 can update the active template with the relevant templateelements that describe the image findings, annotate the image, and/orappend the auto annotated image to the report.

The system 100 can automatically evaluate the consistency between themeasurements made in the viewer and the finding/pathology made in theactive template. For example, if user measures the AC distance to be 4mm and selects an active template element that says there is an ACseparation, then the system 100 can warn the user, e.g., through apopup, that the measurement and the selected active template element arein consistent. On the other hand, if the user measures the AC distanceto be larger than 6 mm and chooses presence of AC separation, ormeasures the AC distance to be smaller than 6 mm and chooses absence ofAC separation, the system 100 would not warn the user of any discrepancybetween the distance measurement and the choice of active templateelement. Typically, the measurement ranges associated with differenttemplate element choices are defined by subject matter experts.

Reports generated using the active template(s) can have standardizedformats and can make communications, e.g., among different entities orindividuals, easy. A user can leverage the information providedautomatically by the viewer, such as the viewer 400 of FIG. 4, and thedecision support system to generate a report. A user can generate oraccess a report associated with an active template directly from theactive template. In the example of the template 500 of FIG. 5, a usercan activate the report button 505. In the example of the template 570,a user can activate the show report text button 582.

A report stored in the system 100 can have different statuses, includinga new status in which a report is not generated or completed; asubmitted status, in which the report is completed by an initialreviewer; a verified status in which the report is reviewed in astandard radiology report format, or text format; a dictated statusadvanced from the verified status, in which the report is noted in thesystem 100 as being ready for review by a second, advanced reader forquality assurance; a preliminary status in which the report is updatedby the quality assurance reader; and a final status, in which the reportis ready for delivery to a client. The report statuses can be altered inboth the advancing direction and the backward direction. As an example,the initial reviewer can be a beginner and the generated report can bein the verified status. The initial reviewer can then choose to advancethe verified report to a dictated status to notify the system 100 thatthe report is ready for review by a second, advanced reader for qualityassurance. The advanced reader then changes the status of the reportinto preliminary. Revision of the standard text version of the reportcan be toggled back to the template version effortlessly so that asecond reader can review the exact choices made by the initial readerand easily modify the report potentially by a combination ofde-selection of choices and modifiers as well as selection of previouslyunselected choices and modifiers. Similarly, revision of the templateversion of the report can automatically revise the text version of thereport. The status of each report for each case is listed in theworklist 300 of FIG. 3, as status 330.

A report can be displayed in different forms on a user interface to auser. The forms can include the active template form which includescolor-coded user selections of active template elements, a text formatwhich includes a non-editable text output of the active template withthe user selections, and a free text format. The template form and thetext format are generally independent of the status of a report. In someimplementations, the free text format is available only in thepreliminary status. In the free text format, a user is allowed toperform additional review and to edit the report. A report in eachdifferent status can be shown in multiple different formats. Forexample, when a report is in the verified, the dictated or thepreliminary status, the report can be toggled between the activetemplate format and the non-editable text output display. In someimplementations, a report in the final status is non-editable and shownin the text format.

An example of a preliminary report 800 is shown in FIG. 8A. The report800 can be viewed in an interface 802 and can contain patientinformation 804, e.g., in the form of a DICOM header. The report 800also contains report information 806, which includes the current status803 of the report and findings 805 in text. A user can save or changethe current status 803 of the report using buttons 807. The report 800can also contain study thumbnails 808, the expansion of which shows oneor more images of the patient for whom the report is generated. Asdescribed previously, a user can drag and drop the image(s) into thereport. An authorized user, e.g., a user with a desired level ofexperience, can verify the report, e.g., by reviewing choices made inthe associated template(s). The authorized user can be the user whogenerated and submitted the report, or a different user when the usergenerated and submitted the report is a trainee.

FIG. 8B shows an example of a report 810 in the verified status in thetemplate mode displayed in an interface 812. The report 810 containspatient information 814 and report information 816, both of which beingin the form of a DICOM header and similar to those contained in asubmitted report. The report 810 also contains findings 818 of the studyor diagnosis. However, instead of being in text, the findings 818contain selected template elements from a corresponding active template.The findings 818 contain all categories of the active template that arerelevant to the study and the chosen template elements. In someimplementations, some findings are critical findings, i.e., timesensitive in patient care, and the system 100 can send autonotifications of such critical findings to the patient's primary medicaldoctor, hospital, RN (registered nurse), or the patient himself/herself.Similar to those categories and template elements in a template, thecategories and template elements in the verified report can be edited,new categories and template elements can be added, some templateelements can be nested. In addition, a user can access the templateselection interface or the decision support system interface through thereport 810, e.g., by activating buttons 820, 822, 824. The verifiedreport 810 also contains the same study thumbnails 826 as itscorresponding submitted report.

In some implementations, when the active template is incorporated in areport window, e.g., the report window 574 of FIG. 5C, the activetemplate displays all selected template elements. For the purpose ofdiscussion, the report 810 is used as an example. The descriptions aresimilarly applicable to the report of FIG. 5C.

The report 810 additionally includes a communication box 830 which isfound in all statuses of the report and can be used by anyone involvedin the acquisition through report review and beyond to communicate withanyone involved in the imaging chain/life cycle of the study report. Forexample, it can contain communications from a client technologist notingpatient history or issues related to the ability to do a quality studysuch as patient motion, or immobility through to feedback from asurgeon, a lab, a clinician, or a patient which can be used to notifythe system of independent information related to the diagnosis/diagnosessuggested by the Final report. The feedback can include opinions such asagree or disagree, and/or data entry if specific surgical, laboratory,pathology correlation is performed.

In some implementations, the active template(s) logs all choices a usermakes and the system 100 can provide feedback to the users on anindividual basis using the logged choices. For example, thediscrepancies between choices made by a less experienced user and thosemade by a more experienced user can be recorded and provided to the lessexperienced user. The less experienced user can learn through thisprocess and improve his/her skills.

In some implementations the sequence of user choices can be shown in aplayback mode so that a beginner user or an advanced user can evaluatethe process as performed by the other. In addition the time spent oneach choice can be identified to determine the time spent on eachHeader/Category. The identified time can be used to help identify areasof improvement.

In some implementations, the system 100 can provide real time feedbackon accuracy of user choices or diagnosis directly in the activetemplate. The feedback can be triggered when the user has reached apredetermined miss rate threshold. For example, a user who keeps missingrenal cysts would start to see the color of the “renal cyst” templateelement in the active template turn yellow as a caution indication. Theuser is alerted to specifically evaluate the choice. Sometimes the usercan even be required to choose positively or negatively whether aspecific finding is present or absent before submitting a report basedon exceeding a threshold. In another example, real-time feedback caninclude pop-ups of the miss rate threshold and its related warnings andguidance.

In some implementations, the system 100 is used in a dual read,teaching, or learning mode, in which feedback is provided to a beginneror a learner automatically by the system 100 and/or by a moreexperienced user. Some features related to the dual read or feedbackhave been discussed previously. In addition, a user's diagnosis processor report can be examined and evaluated by another user, e.g., a moreexperienced user or a quality assurance user, or the system 100. Theexamination can be performed in resident training of the user. Thesystem 100 may allow the more experienced user or quality assurance userto study the record of the patient based on which a trainee made his/herdiagnosis or report. The experienced user can change the existingdiagnosis or report of a less experienced user through the activetemplate(s). The less experienced user can be evaluated based on thedifferences between the choices made by the two users in the activetemplate(s). The system 100 can then generate a mistake score for thetrainee to reflect a discrepancy between the trainee's diagnosis orreport and that of the more experienced user.

In some implementations, the more experienced user or quality assuranceuser can score a trainee's diagnosis process or report by reviewing thestored viewer, template(s), and report(s) based on which the traineemade his/her diagnosis or report. In some implementations, the system100 allows a user to use the system in a playback mode, in which theuser can play back the time and sequence of each mouse click. For eachtrainee, the system 100 can automatically accrue a lesion conspicuityscore and clinical severity score when a more experienced user uncoversa miss in the diagnosis or the report, without the more experienced userdoing any additional work. For example, the findings or templateelements of an active template, e.g., the active template 574 of FIG.5C, can also contain attributes such as “Severity” and “Conspicuity”.The values of these attributes can be automatically set by the system100 when a subject matter expert modifies a beginner's choices. Thesystem 100 can generate a grade for the beginner as a byproduct of amore experienced user correcting the report choices.

The scores/grades for users can be shown using features of the RIS(Radiology Information System). For example, field filter function canbe applied on a worklist for a specific trainee. Depending on thespecific question used in the filtering, an overall score for thetrainee or a specific score based on evaluations on a specific modalitybody part can be retrieved. The system 100 can also rank the trainees.For example, the system 100 can automatically correlate a trainee's ranknumber, e.g., compared to either norms or a cohort, with severity ofmisses, his/her conspicuity score, and the user efficiency related timestamps, i.e., time spent on a study. The system 100 can alsoautomatically score the users based on the correlations to reflect theusers' efficiency and accuracy in classifying normal and abnormalfindings. Conspicuity scores associated with certain pathology maydepend on the features of the pathology. For example, when the pathologyis a lesion, the conspicuity score is related to the size of the lesionand the conspicuity attribute can have a size range associated with aspecific conspicuity score.

In some implementations, choices made in the active template(s) andimages dragged into the active template(s) populate the decision supportsystem with examples. For example, the system 100 can enable the user toidentify through the interface(s) displaying the template(s) and/or thedecision support system, one or more choices or images as representingan example for the decision support system in an appropriate study. Oneor more moderators can verify whether the examples are appropriately andcorrectly listed in the decision support system. In someimplementations, a user's drag and drop action triggers a moderator'sreview.

Decision Support System

As described previously, a user can access a decision support systemthrough other interfaces, e.g., a viewer interface or an active templateinterface. An example decision support system 700 is shown in FIG. 7. Adecision support system interface 702 displays the support system 700 toa user. The decision support system 700 can aid the user with relevantinformation and content for generating a report based on informationfrom the viewer and from the active template. The system 700 canleverage information which is already known in the field as a byproductof the users' viewing the images and interacting with the activetemplate(s) so that relevant content of the information is provided tothe user at the time and point of contact with the user.

Similar to the template elements that have attributes, the decisionsupport system also contains elements that support data attributes,which can be static or variable, and function attributes. In someimplementations, the elements in the decision support system can betagged with unlimited attributes. The tagging can be related to contextsensitive presentations of the elements to the user, and/or the behaviorof the attributes, e.g., when the elements are dragged into a report orclicked on to produce changes in either the active template(s) or theviewer.

Example elements in the decision support system include images,diagrams, and texts. For example, in a part of the decision supportsystem related to calcaneus classification, actual examples of calcaneusfractures with text elements noting key features of the various fracturetypes are provided. Each of the elements, which can be X-ray images,diagrams, descriptive texts, and others, can activate active templatefunctions, including inserting additional template elements into anactive template for use in describing the calcaneus fracturesillustrated in the elements. Some attributes ascribed to the elements,including calcaneus diagrams, images, and texts, can activate or triggerspecific viewer tools, e.g., the Boehler's angle measurement tool.

In the example shown in FIG. 7, the decision support system 700 isinitially displayed to a user in the form of an index in a window 704that shows a navigation panel. Support elements are hidden under theindex and can be accessed when the user reaches the lowest level of theindex. When the support elements contain data support attributes, thedata can be displayed in another window 706. For example, the supportelements contain examples of normal, abnormal, and normal variants,which can be shown in the window 706. In addition, the support elementsperform the function of informing active templates of relevant“choice/modifier template elements, and/or informing the viewer whichtool(s) to use and how to use it.

In some implementations, the index is arranged similarly to the templatenames shown in FIG. 6A. The highest level 708 of the index sorts thesupport elements based the medical methods used in obtaining the medicalrecords or images. For example, “CT” represents CT scan, “DX” representsDiagnostic X-ray, “MR” represents MRI scan. The next level 710 of theindex sorts the support elements based on the body parts on which themedical methods are applied. The next level 712 further sorts thesupport elements based on various parameters associated with the medicalmethods applied to the body part. If necessary, additional levels areadded to sort the support elements. In some implementations, multipleuser interfaces of the system 100 are displayed simultaneously to auser. To free up display space, e.g., desktop space, the navigationpanel shown in the window 704 can be closed or collapsed in its defaultstate.

The correspondence between the template names and the index arrangementcan help a user quickly familiarize himself/herself with the system 100and readily locate the support information needed from the supportsystem 700. In some implementations, the system 100 links one or moretemplate elements of an active template with one or more supportelements of the support system 700, such that when a user opens adecision support system from a link nearby a linked template element,the system 100 directly displays to the user the linked supportelements. In some implementations, instead of displaying the linkedsupport elements, the index level 710 or lower associated with the bodypart studied in the active template is displayed. In the example shownin FIG. 7, when a user studying chest results of a patient and accessesthe decision support system 700, the index level 712 corresponding tothe chest is displayed to the user.

A user can navigate through the decision support system 700 using theindex. In some implementations, a user's choices made in an activetemplate can be automatically populated to the support system 700 suchthat corresponding support information is automatically displayed,simultaneously to the user's actions on the active template.

In some implementations, the decision support system provides differentaccess permissions to different users. For example, all users who areusing the decision support system for classifying can access the fullsystem. The support system 700 can be presented to the user in a guidedmode or an expert mode, similar to the guided mode or the expert mode ofthe template described previously. In the guided mode, thecategory/Header which a user, e.g., a learner, beginner, or trainee, isreviewing in the active template has an automatic link to a relevantpage in the decision support system. The link can be predetermined by asubject matter expert when building the active template. In the guidedmode, the decision support system follows the user's activity on acorresponding active template in a step by step manner, presentingdiagrams, discussions, and examples of normal, abnormal, and normalvariant information in the window 706. In the expert mode, the decisionsupport system follows the user based on the user's clicking of thechoice and modifier template elements. Experienced users, e.g., experts,can navigate through the decision support system by activating specificdecision support system link icon(s) at their discretion.

An authorized user of the support system 800 can edit the contents ofthe support system through an edit button 720 in the interface 702. Inaddition, the authorized user may also create one or more new supportelements under proper index level to include knowledge that is notcurrently stored in the system 800. In this example, the user can createsuch new elements through a new button 722 on the interface 702.

The decision support system 700 can also include example case pageswhich contain thumbnail images demonstrating common pathologicpresentations and descriptions of the images by one or more subjectmatter experts. The images can help a user in diagnosing, and thedescriptions can be used by the user in the report so that the user canbe efficient and the report can be standardized. For example, intuberculosis chest x-ray reports, there are common and characteristicpresentations based on stage/severity/chronicity and the descriptionscan be done by a subject matter expert and incorporated into the exampleimages on the decision support system as attributes. When an exampleimage is dragged into the active template at the appropriate spot, theattributes are inserted into the active template as new templateelements. All reports on the same diagnosis can use the descriptionsconsistently.

In some implementations, based on the support elements, examples, anddiscussion information of the decision support system, a user cancustomize, edit, or make new active templates. For example, additionaltemplate elements can be added or some template elements can bemodified, based on pre-classified findings recorded in the supportsystem. In addition, known image findings can be incorporated into thesupport system 700, e.g., an image showing a particular diagnosis can bedragged and dropped into the window 706 that corresponds to the supportelement for the diagnosis. An expert or administrator authorized in thesupport system can add additional description or comments to the newlyadded image findings, including the texts used in the active templateelements that describe the findings as attribute of an element of thenewly dragged and dropped image in the decision support system. Thecommunication between the template, the viewer, and the decision supportsystem can promote best practices for the classification and itspotential variability.

Object Oriented Programming

In some implementations, one or more of the tools in viewers, thetemplate elements in active templates, the elements in the decisionsupport system, including diagrams, images, text blocks, etc., areencoded in a method similar to object oriented computer program code.The attributes of these tools or elements are not limited and caninclude those initially not contemplated and later added. The attributesare expressed as attributes of objects, where the previously discussedelements in the decision support system, active templates, and viewerscan be the analog to objects in computer coding.

In some implementations, values of the attributes initially set in thesystem 100 may be changed during the use of the system 100. For example,the norms or normal ranges encoded in the system 100 can be adjustedbased on actual data generated from use of the system 100. Image dataand its corresponding findings can be aggregated for generating moreaccurate normal ranges. In the example of AC distance, instead ofsmaller than 6 mm being normal as set initially by a subject matterexpert, the aggregated data may show that the normal range may be lessthan 5.43 mm. Values of these attributes can also be changed in responseto actions by a user in using one or more of the interfaces.

In some implementations, the objects (or elements) of the decisionsupport system, active templates, and/or the viewers, can haveattributes that are internet links to content external to the system100. The content can include peer reviewed journals or book chaptersthat support the use of specific values for the determination of thestate of a subject. Elements in the active templates, the decisionsupport system, and the viewer can also link to other pages. Forexample, within a page of a decision support system, there can beinternal links to other pages of the same system. In another example,using icons similar to the link 548 of FIG. 5, a user can link from anactive template to the decision support system. Sometimes activation ofan element of the system 100 can automatically cause navigation to aspecific decision support page.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on an artificiallygenerated propagated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable digitalprocessor, a digital computer, or multiple digital processors orcomputers. The apparatus can also be or further include special purposelogic circuitry, e.g., an FPGA (field programmable gate array) or anASIC (application specific integrated circuit). The apparatus canoptionally include, in addition to hardware, code that creates anexecution environment for computer programs, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, anoperating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a data communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). For a system of one or morecomputers to be “configured to” perform particular operations or actionsmeans that the system has installed on it software, firmware, hardware,or a combination of them that in operation cause the system to performthe operations or actions. For one or more computer programs to beconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions.

Computers suitable for the execution of a computer program include, byway of example, can be based on general or special purposemicroprocessors or both, or any other kind of central processing unit.Generally, a central processing unit will receive instructions and datafrom a read only memory or a random access memory or both. The essentialelements of a computer are a central processing unit for performing orexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of non volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

Control of the various systems described in this specification, orportions of them, can be implemented in a computer program product thatincludes instructions that are stored on one or more non-transitorymachine-readable storage media, and that are executable on one or moreprocessing devices. The systems described in this specification, orportions of them, can be implemented as an apparatus, method, orelectronic system that may include one or more processing devices andmemory to store executable instructions to perform the operationsdescribed in this specification.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of any claimsor on the scope of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A computer based method comprising displayingsimultaneously to a user: (a) at least one current image of a subjectand at least one tool that enables the user to make a measurement on thecurrent image, the image being associated with a subject matter, (b)decision support information to aid a user to determine and to express adetermination about a state of the subject based on observation of andmeasurement by the user of the current image, the decision supportinformation including a representative figure that corresponds to theimage and that a human who is an expert on the subject matter haspreviously determined graphically depicts one or more states thatgenerally characterize a population of subjects with respect to thesubject matter, the decision support information including proseprovided by a human who is an expert on the subject matter, describingthe states that generally characterize the population of subjects andhow to make a determination about the state of a subject, and (c)interactive prose elements available for use by the user to express thedetermination about the state of the subject, each of at least some ofthe interactive prose elements being invocable by the user to alter thedisplay with respect to at least one of (a) the number of prose elementsthat are available to express the determination, (b) the identity of theprose elements that are available to express the determination or (c) anexplanation of one or more of the prose elements, the interactive proseelements being predetermined by a human who is a subject matter expertto be usable to correctly and completely express the determination, atleast two of the image, the decision support information, and theinteractive prose elements being displayed so that actions of the userwith respect to one of the two causes changes in the other of the two,the representative figure included in the decision support informationbeing associated with a predefined set of words from which a subset ofwords can be chosen by a user who is observing the current image todescribe correctly and completely the state of the current subject, thepredefined set of words having been chosen by a human who is an experton the subject matter, in response to the user dragging therepresentative figure relative to the interactive prose elements,automatically updating the displayed interactive prose elements toinclude the predefined set of words chosen by the human who is an experton the subject matter, and in response to the user making a measurementon the image using one of the tools, automatically including with theinteractive prose elements, a prose expression of a result of themeasurement.
 2. The method of claim 1 in which the interactive proseelements represent templated structured content and interaction with theinteractive prose elements is reflected in a digitally stored recordassociated with the subject.
 3. The method of claim 1 in which the usercan manipulate the image to visualize image findings or makemeasurements of the image features or both.
 4. The method of claim 3 inwhich the manipulating of the image is context-sensitive.
 5. The methodof claim 4 in which the context in which the manipulating iscontext-sensitive comprises an imaging modality or a part of the subjectthat is imaged or both.
 6. The method of claim 1 in which theinteractive prose elements represent selections, or choices, ormodifiers, and interaction with the interactive prose elements governsat least a part of the determination that is expressed about the stateof the subject.
 7. The method of claim 6 comprising displaying to theuser of the browser at least one tool that aids the user inunderstanding the images, the tool being contextually sensitive to userinteraction with the interactive prose elements.
 8. The method of claim7 in which a use of the tool causes at least one value associated withthe tool to be associated with a feature of the interactive proseelements.
 9. The method of claim 8 in which the use of the toolcomprises a dragging and dropping of an image to which the tool has beenapplied.
 10. The method claim 1 in which at least two of the image, thedecision support information, and the interactive prose elements areassociated with stored attributes associated with images, interactiveprose elements, or decision support elements, and at least some of theattributes are expressed as objects that include values and methods. 11.The method of claim 10 in which values of at least some of theattributes are changed in response to actions by the user.
 12. Themethod of claim 1 in which, if information entered by the user isindicative of a critical finding that requires time sensitive action,automatic notification is sent to a party for action.
 13. The method ofclaim 1 in which displaying the interactive prose elements includesdisplaying, in response to the user interaction with one or more of theinteractive prose elements information that guides the user in selectinginteractive prose elements for inclusion in a report about the state ofthe subject.
 14. The method of claim 1 comprising altering interactiveprose elements to indicate to a particular user an aspect of theexpressed determination that requires attention by that user.
 15. Themethod of claim 1 in which the user is provided real-time feedback basedon information about the user, the subject, or the expresseddetermination.
 16. The method of claim 15 in which the real-timefeedback comprises notifications about errors made by the user.
 17. Themethod of claim 15 in which the real-time feedback comprisesnotifications associated with syndromes or common co-morbiditiessuggested by the user's capturing of certain expressed determinationabout the state of the subject.